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ABSTRACT 


During  October  2008,  an  Air  Force  Research  Event  was  conducted  to  integrate 
operational  concepts  and  training  techniques  from  different  commands.  The 
collaborative  teamwork  demonstrated  in  the  highly  asymmetric  threat  exercise  scenario 
was  recorded  in  Mardem-Bey  internet  relay  chat  logs  across  fifteen  different  chat  rooms. 
The  goal  of  this  thesis  was  to  use  chat  room  recorded  data  that  was  extracted  from  an 
exercise  environment  simulating  an  air  and  space  operations  center  (AOC)  cell  using  an 
experimental  structure;  non-approved  tactics,  techniques,  and  procedures;  and  a  fictional 
environment  to  stimulate  certain  training  processes  to  evaluate  a  measurement  model  of 
macrocognition  developed  under  the  Collaboration  and  Knowledge  Integration  program, 
sponsored  by  the  Office  of  Naval  Research.  The  model  focuses  on  team  member’s 
cognitive  processes  used  during  collaboration  with  the  goal  of  trying  to  understand  how 
individuals  collaborate  to  build  new  knowledge  and  accomplish  their  tasks.  Effective 
chat  communications  may  expedite  the  process  of  moving  the  team  towards  achieving 
their  ultimate  goal,  which  was  to  produce  optimum  combat  effectiveness  in  a  timely 
manner.  Thesis  results  will  be  provided  to  the  Office  of  Naval  Research  to  help  improve 
collaboration  among  teams  while  operating  in  stressful  and  dynamic  environments. 
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I.  INTRODUCTION 


A.  THESIS  GOAL 

1,  Collaboration  and  Knowledge  Interoperability 

The  Department  of  Defense  is  being  confronted  by  a  highly  capable  and  quickly 
adaptable  adversary.  The  adversary  fights  in  a  constantly  changing  and  dynamic 
environment  using  asymmetrical  warfare  techniques.  In  order  to  effectively  combat  and 
defeat  such  an  enemy,  human  decision  makers  at  the  tactical,  operational  and  strategic 
levels  must  act  and  decide  rapidly.  To  enable  personnel  to  make  quicker  and  more 
accurate  decisions,  personnel  at  all  levels  of  the  chain  of  command  must  be  given  access 
to  more  detailed  information  than  ever  before  and  collaborate  with  several  inter-theater 
and  cross  border  multinational  and  multiagency  personnel. 

The  Office  of  Naval  Research  (ONR)  sponsored  Collaboration  and  Knowledge 
Interoperability  program  attempts  to  gain  insight  and  understanding  into  the  human 
cognitive  processes  demonstrated  during  an  individual’s  (or  group  of  individuals)  attempt 
at  solving  extremely  complex,  time  sensitive,  and  dynamic  problems.  ONR  studies 
suggest  that  by  gaining  greater  insight  and  understanding  of  all  the  metacognitive  and 
macrocognitive  processes  exhibited  by  humans  during  the  entire  decision  making 
process,  better  system  architectures,  models  and  methods  can  be  developed  to  improve 
the  reliability  and  speed  of  data  dissemination  to  the  human  decision  maker. 
Additionally,  creation  of  human  agent  interfaces  based  on  human  factor  principles  would 
provide  better  information  displays  to  increase  user  understanding  and  ease  of  system  use 
therefore  improving  effectiveness  of  decision  making  (Letsky,  2008). 

ONR  developed  a  model  of  team  collaboration  and  cognitive  processes  (Figure  1) 
that  includes  the  input  factors  (time  pressure,  information  uncertainty,  dynamic 
environment  factors,  etc...),  and  phases  (knowledge  construction,  collaborative  team 
problem  solving,  team  consensus,  and  outcome  evaluation  and  revision)  demonstrated 
during  team  collaboration. 
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Collaboration  Stages  &  Cognitiva  Pmntutsaie 
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•  team  negotiation  of  solution 
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•  team  pattern  recognition 

•  convergence  of  individual 
mental  models 

•  critical  thinking 

•  sharing  hidden  knowledge 
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Figure  1.  Collaboration  Stages  and  Cognitive  Proeess  Model  (From  Warner,  Letsky,  & 

Cowen,  2005). 

The  Model  of  Team  Collaboration,  shown  in  Figure  1,  allows  researchers  the 
ability  to  try  and  predict  human  decision-making  outcomes  in  specific  real-world  events 
and  exercise  scenarios  (Warner,  Letsky,  &  Cowan,  2005). 

2.  Goals  for  the  SUMMIT  MURI  Research 

In  October  2008,  an  evolutionary  change  to  the  original  Model  of  Team 
Collaboration  was  suggested  during  the  Office  of  Naval  Research  Systems  for 
Understanding  and  Measuring  Macrocognition  in  Teams  (SUMMIT)  meeting  sponsored 
by  a  Multidisciplinary  University  Research  Initiative  (MURI)  grant.  The  evolutionary 
changes  were  needed  to  address  human  related  collaboration  issues  that  have  been 
discovered  as  the  Department  of  Defense  steadily  increases  its  reliance  and  dependence 
on  network  centric  warfare.  Network  centric  warfare  requires  strong  communication  and 
detailed  coordination  among  all  operational  and  tactical  units  located  within  or  outside 
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the  operational  theater.  For  network  eentric  warfare  to  be  suecessful  several 
geographically  separated  operational  and  tactical  personnel  must  be  able  to  quickly  form 
into  a  cohesive  team.  Despite  possible  individual  biases,  the  team  must  be  able  to 
effectively  share  and  process  information  (convert  data  to  information  and  information  to 
knowledge)  in  order  to  facilitate  faster  decision  making  and  effectively  combat  the 
enemy.  The  MURI  team  conducted  detailed  research  and  investigated  how  the  delivery, 
availability  and  different  types  and  information  and  information  processing  systems 
affected  the  macrocognitive  processes  within  teams  that  were  both  centrally  located  and 
distributed.  The  MURI  research  goal  is  to  improve  collaboration  process  understanding 
and  creation  of  better  tools  to  better  support  macrocognition  in  teams.  Additionally,  the 
research  focused  on  creating  a  test  environment  and  produced  network  centric  warfare 
cognitive  metrics  that  can  be  used  to  evaluate  team  cognitive  behavior  (Fiore,  Rosen, 
Salas,  Burke,  Warner,  &  Letsky,  in  press). 

3.  Major  Goals  for  This  Thesis 

The  major  goal  of  this  thesis  is  to  evaluate  the  new  Office  of  Naval  Research 
sponsored  Measurement  Model  for  Macrocognition  Research  using  the  new  SUMMIT 
coding  definitions,  dated  October  2008.  Definitions  of  the  macrocognitive  processes 
included  in  the  model  were  applied  to  the  Research  Event  Air  and  Space  Operations 
Center  Dynamic  Effects  Cell  generated  data.  Eigure  2  is  a  graphical  representation  of  the 
measurement  model  metacognitive  phases  and  illustrates  how  individuals  functioning  as 
a  team  process  and  learn  new  information.  The  new  SUMMIT  coding  definitions  will  be 
discussed  in  detail  in  Chapter  IV  of  this  thesis  and  the  coding  process  will  be  explained  in 
Chapter  V  of  this  thesis. 
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Legend: 


Output(s) 


Figure  2.  Measurement  Model  for  Maeroeognition  Researeh  (After  Fiore  et  al.,  in 

press). 


The  Oetober  2008  Researeh  Event  provides  us  with  a  realistic,  dynamic,  time- 
sensitive  situation  to  assess  and  validate  the  measurement  model.  By  coding  the  data 
from  the  exercise  utilizing  to  the  new  SUMMIT  definitions,  we  can  determine  if  the 
metacognitive  processes  shown  in  the  measurement  model  (Figure  2)  are  an  actual 
representation  of  how  teams  collaborate  to  handle  dynamic  problems  in  a  time  sensitive 
environment. 

During  coding,  it  was  discovered  that  there  were  existing  gaps  in  the 
measurement  model  and  the  SUMMIT  definitions.  Theses  gaps  are  explained  in  Chapter 
VI,  results  section  of  this  thesis,  along  with  recommendations  to  the  definitions. 
Additionally,  some  of  the  SUMMIT  definitions  were  not  used  during  our  coding  of  the 
Research  Event  data.  A  brief  explanation  of  why  certain  codes  weren’t  used  is  included 
in  Chapter  VI. 
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4. 


Goals  for  the  AOC  Dynamic  Effects/Targeting  Cell 


The  main  goals  of  the  Air  Foree  Research  Event  are  listed  in  Table  1.  For  the 
purpose  of  this  thesis  the  terms  Dynamic  Effects  Cell  (DEC)  and  Dynamic  Targeting  Cell 
(DTC)  are  synonymous. 


Table  1 .  Air  Force  Research  Event  Dynamic  Effects  Cell  Research  Objectives 

(From  Air  Force  Research  Laboratory  Warfighter  Readiness  Division,  2008). 


Air  Force  Research  Event  Dynamic  Effects  Cell  Research  Objectives 


Goal  #1 :  Assess  effects  of  chat  protocol  on  communications  and  mission  performance: 

-  Measure  effectiveness  of  chat  room  user 

-  Measure  of  efficient  times  from  info  drop  to  engagement 
Goal  #2:  Highlight  and  explore  future  concepts  for  continuous  learning: 

-  Use  individual  training  sessions  to  train  in  Air  and  Space  Operations  Center 
environment. 

-  Assess  war  fighter  planning,  employment  and  effectiveness 

Goal  #3:  Integrate  war  fighter  controllers  and  information  simulation  exercise 
participants: 

-  Build  and  employ  Master  Scenario  Event  Eist  and  Request  For  Information 
management  environment. 

-  Engage  Senior  Intelligence  Duty  Officer  and  Target  Duty  Officer  with  realistic 
interactions  and  survey  for  effect 

Goal  #4:  Use  of  multiple  performance  measurement  systems 

Goal  #5:  Assess  effects  of  lED  Network  Defeat  Tactics,  Techniques  and  Procedures  for 
non-kinetic  mission  execution 


This  thesis  evaluated  the  team  collaboration  within  the  Air  and  Space  Operations 
Center  (AOC)  DEC,  which  is  a  contributing  factor  for  measuring  the  effectiveness  of  all 
the  Air  Force  Research  Event  research  goals. 
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B,  AIR  AND  SPACE  OPERATIONS  CENTER  DYNAMIC  EFFECTS/ 

TARGETING  CELL 

1.  Mission 

After  five  years  in  Afghanistan  and  Iraq,  the  United  States  and  Coalition  forees 
remain  actively  engaged  against  a  highly  capable  and  adaptive  enemy.  Unlike  the  Cold 
War  standoff  with  the  Soviet  Union  that  matched  nation  against  nation,  massive  force 
against  massive  force,  and  military  technology  against  military  technology,  today’s 
enemy  does  not  fight  for  a  particular  country,  has  no  visible  army  to  call  its  own,  and  has 
very  limited  technology,  capabilities  and  resources  to  combat  coalition  forces.  The 
enemy  does  however  have  a  very  strong  will  to  fight  and  the  means  by  which  to  use 
irregular  warfare  techniques  to  create  mass  fear  and  to  disrupt  the  life  of  those  friendly  to 
the  United  States’  way  of  life.  The  enemy’s  employment  of  improvised  explosive 
devices  (lEDs)  has  caused  massive  fear,  severe  damage,  and  the  loss  of  life  of  several 
sailors,  soldiers,  and  countless  Iraqi  civilian  personnel.  To  counter  the  lED  threat  and 
take  advantage  of  the  limited  targeting  window  provided  by  those  responsible  for  lED 
employment,  new  ways  have  been  adapted  by  the  AOC  to  quickly  and  accurately  conduct 
dynamic  targeting  against  the  enemy  and  lED  threats. 

The  employment  of  the  dynamic  effects/targeting  cell  process  against  lED 
networks  is  currently  being  researched  by  the  Air  Eorce  Research  Eaboratory  in 
collaboration  with  the  USAE  Warfare  Center.  Adaptation  of  a  dynamic  effects  cell 
within  the  AOC  (Joint  and  Combined)  command  structure,  Eigure  3,  would  provide  the 
Joint  Eorce  Air  Component  Commander  (JEACC)  the  leverage  to  engage  and  attack 
targets  of  opportunity  that  meet  Joint  Eorce  Commanders  mission  objectives.  Dynamic 
targeting  is  a  means  by  which  coalition  forces  could  possibly  respond  to  the  employment 
of  lEDs  and  insurgent  network  leadership  located  within  Afghanistan  and  Iraq.  Due  to 
the  short  time-sensitive  nature,  dynamic  targets  are  normally  vetted  quickly  and 
accurately  through  the  targeting  cycle.  This  process  allows  the  JEACC  to  task  available 
assets  or  reassign  assets  to  engage  and  destroy  known  targets  or  other  potential  threats  at 
a  moment’s  notice.  The  limited  time  window  available  for  target  engagement  and 
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destruction  means  that  coalition  forces  must  be  prepared  to  apply  timely  and  accurate 
measures  and  counter-measures  against  the  enemy. 

2,  Dynamic  Effects  Cell  Area  of  Responsibilities 

The  DEC  would  be  responsible  for  directing  the  planning  and  coordination  of  all 
dynamic  targets,  time-sensitive  targets  (TSTs),  and  high  payoff  target  operations  (U.S. 
Air  Force  Central  Command,  no  date).  Currently,  highly  trained  personnel  and  the 
advances  in  intelligence,  surveillance  and  reconnaissance  (ISR)  assets  provide  the 
necessary  tools  to  find,  fix,  target,  track,  engage,  and  assess  targets  of  opportunity  that 
require  immediate  attention  and/or  action.  The  airborne  warning  and  control  system, 
joint  surveillance  targeting  attack  radar  system  and  unmanned  aerial  vehicles  continue  to 
provide  the  Commander  of  U.S.  Central  Command  and  the  Combined  Force  Air 
Component  Commander  (CFACC)  the  means  to  destroy  TSTs,  such  as  lEDs,  key 
insurgent  leadership  personnel,  and  terrorist  network  cells.  Of  the  3500  targets 
nominated  by  the  CFACC  during  Operation  Iraqi  Freedom,  156  were  classified  as  TSTs 
and  686  were  identified  as  dynamic  targets  (Moseley,  2003). 

3.  Typical  Operational  Components  and  Players 

The  DEC,  if  employed  operationally,  would  fall  within  the  Combat  Operations 
Division  of  the  AOC.  Dynamic  target  nomination  is  coordinated  with  the  combat 
operations  division  via  the  Senior  Offensive  Duty  Officer  (SODO).  The  organization 
makeup  depends  on  the  overall  operational  requirements  and  Joint  Force  Commander 
(JFC)  mission  objectives.  If  included  as  a  component  of  the  AOC,  the  DEC  at  a 
minimum  would  include  a  DEC  Chief,  a  Deputy  DEC  Chief,  a  Ground  Track 
Coordinator,  an  Attack  Coordinator,  and  a  Target  Duty  Office  as  shown  in  Figure  3  (U.S. 
Air  Force  Central  Command,  in  press).  Proper  synchronization  of  ISR  collection  support 
for  target  tracking  and  engagement  between  the  Senior  Intelligence  Duty  Officer  (SIDO) 
and  DEC  would  be  vital  for  overall  AOC  mission  success  and  for  de-confliction  of 
friendly  fires. 
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Figure  3.  Dynamic  Effects  Cell  Organizational  Structure  (From  U.S.  Air  Force  Central 

Command,  in  press). 

4,  AOC  Organization  and  Responsibilities 

Today’s  AOC  is  the  focal  point  from  which  planning,  directing,  controlling  and 
coordinating  of  air  and  space  operations  comes  together  to  satisfy  the  JFC  objectives. 
The  AOC  must  be  prepared  to  carry  out  deliberate  dynamic  targeting  without 
compromising  its  capability  to  conduct  other  major  air  combat  operations  (U.S.  Air  Force 
Central  Command,  no  date).  According  to  the  Air  Force  Instruction  13-1  AOC  Volume 
3,  a  typical  AOC  command  structure  will  consist  of  an  AOC  Director,  five  divisions 
(Strategy,  Combat  Plans,  Combat  Operations,  Intelligence  Surveillance  and 
Reconnaissance,  and  Air  Mobility),  and  multiple  support  specialty  teams.  Although  not 
all  inclusive,  such  a  command  structure  is  extremely  vital  to  AOC  mission  success. 

5,  Joint  Force  Air  Component  Commander  Responsibilities 

The  JFACC  is  assigned  by  the  JFC  to  manage  the  air  war  in  theater.  The  JFACC 
is  responsible  for  effective  planning,  coordination,  allocation,  and  theater  tasking  of 
assets  to  accomplish  the  JFC  mission  (Joint  Publication  3-30,  2003).  The  JFACC 
accomplishes  the  assigned  mission  in  accordance  with  the  guidance  and  under  the 
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authority  granted  by  the  JFC.  The  JFACC  exereises  operational  and  taetieal  control  over 
all  assigned  air  assets  and  personnel  located  within  the  theater  of  operation.  The 
allocation  of  assets  to  destroy  key  enemy  leadership,  suppression  of  enemy  air  defense 
systems,  convoy  protection,  close  air  support,  and  dynamic  targeting  are  just  a  few  of 
many  ways  by  which  the  JFACC  could  use  the  assigned  assets  to  render  an  adversary’s 
method  of  attack  ineffective  and  thereby  minimize  damage  and  coalition  force  and 
civilian  casualties. 

C.  AIR  FORCE  RESEARCH  EVENT 

During  October  2008,  an  intense  Air  Force  Research  Event  was  conducted 
bringing  together  several  key  operational  command  and  control  military  and  civilian 
personnel  from  all  services.  The  event  brought  together  personnel  from  the  United  States 
Air  Force  Warfare  Center,  Naval  Strike  Air  Warfare  Center,  Special  Operations 
Command,  and  United  States  Army  (Air  Force  Research  Laboratory  Warfighter 
Readiness  Division,  2008).  The  purpose  of  this  Research  Event  was  to  assess  tactics, 
techniques  and  procedures  of  operational  command  personnel  performing  kinetic  and 
non-kinetic  dynamic  targeting  in  a  highly  asymmetric  environment. 

The  Research  Event  was  a  simulation  of  a  12-hour  overnight  shift  in  the  dynamic 
effect  cells  of  a  typical  AOC.  The  12-hour  shift  was  broken  into  six  2-hour  time  sections. 
The  operational  players  were  separated  during  the  exercise  by  use  of  false  walls  (i.e., 
room  dividers)  to  simulate  real-world  space  and  time  disparity  of  the  various  joint  service 
personnel. 

Communication  between  the  various  players  was  facilitated  by  the  use  of  the  Joint 
Automated  Deep  Operations  Coordination  System  (JADOCS).  JADOCS  provides  the 
warfighter  with  a  timely,  accurate,  detailed  battlespace  view  for  planning,  coordination, 
and  execution  of  targets.  It  is  a  joint  mission  management  software  application  that 
provides  a  suite  of  tools  and  interfaces  for  horizontal  and  vertical  integration  across 
battlespace  functional  areas  (Raytheon  Company,  2008).  The  Research  Event 
communications  were  recorded  across  fourteen  different  chat  room  channels.  Eigure  4 
identifies  the  chat  rooms  and  specific  personnel  inside  each  chat  room. 
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Figure  4.  Chat  Room  Layout  (From  Air  Force  Research  Laboratory  Warfighter 

Readiness  Division,  2008). 


Each  of  the  Research  Event  operational  players  had  varied  access  and 
responsibilities  in  the  chat  rooms  used  for  the  exercise.  All  the  operational  players  did 
not  have  access  to  every  individual  chat  room.  Some  key  operational  players  were 
designated  as  room  owner  (“R”),  active  participant  (“P”)  and/or  observer  (“O”).  Eigure  5 
is  a  list  of  participants  and  their  associated  responsibilities  in  each  chat  room. 
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Figure  5.  Chat  Room  Assignments  (From  Air  Foree  Research  Laboratory  Warfighter 

Readiness  Division,  2008). 
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II.  BACKGROUND 


Chapter  II  Background  Section  A  of  this  thesis  is  intended  to  bring  the  reader  up- 
to-date  on  the  history  and  types  of  missions  the  Air  and  Space  Operations  Center  (AOC) 
supports  in  the  battle  space.  Section  B  will  advise  the  reader  on  the  internal  structure  of 
the  AOC  and  responsibilities  of  key  personnel.  Section  C  will  inform  the  reader  on  how 
the  AOC  Dynamic  Targeting  Cell  (DTC)  is  designed  to  handle  asymmetric  threats  and 
describe  specifically  how  DTCs  respond  and  prosecute  time  sensitive  targets. 

A,  AIR  AND  SPACE  OPERATIONS  CENTER 

1.  History  Behind  AOC  Conceptualization 

The  11  September  2001  attacks  on  the  World  Trade  Center  and  the  Pentagon 
revealed  that  U.S.  homeland  security  was  extremely  vulnerable  and  lacked  the  procedures 
for  an  effective  coordinated  response  to  combat  such  highly  coordinated  and  dynamic 
attacks.  At  the  time  of  the  9/11  attacks,  North  American  Aerospace  Defense  (NORAD) 
command,  a  joint  endeavor  between  U.S.  and  Canada,  was  responsible  for  air  security 
and  air  sovereignty  defense  for  all  of  North  America.  NORAD ’s  primary  mission  at  the 
time  of  the  terrorist  attacks  only  considered  the  Soviet  Union  as  a  major  threat  and 
therefore  NORAD  only  trained  and  planned  to  combat  and  respond  to  over-the-horizon 
strategic-strike  ballistic  missile  attacks  from  the  Soviet  Union.  An  attack  from  nineteen 
terrorists  using  U.S.  hijacked  airplanes  was  never  anticipated  or  planned  for  which  caught 
NORAD  unprepared  to  respond  appropriately.  Air  Force  General  Myers,  Chairman  of 
the  Joint  Chiefs  of  Staff,  later  defended  NORAD’s  poor  response  to  the  9/11  attacks  by 
stating  that  NORAD  was  not  tasked  or  responsible  for  attacks  originating  from  with 
inside  the  U.S.  but  only  responsible  for  those  attacks  that  originated  from  outside  the  U.S 
(Banusiewicz,  2004). 

According  to  David  Fulghum,  a  Senior  Military  Editor  for  Aviation  Week  & 
Space  Technology,  “a  common  or  single,  integrated  air  picture  would  let  operational 
commanders  or  intelligence  analysts,  for  example,  follow  the  flight  of  a  suspicious 
aircraft  as  it  moves  across  international  borders  or  from  one  theater  to  another.” 
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(Fulghum,  2004,  58).  It  is  expected  that  integration  of  AOCs  will  result  in  reduced 
reaction  times  between  AOCs  and  provide  seamless  pass  off  tasking  from  one  AOC  to 
another  when  disaster  strikes. 

2,  AOC  Types  and  Responsibilities 


A  total  of  23  AOC  sites  have  been  established  on  air  force  bases  (AFB)  located  in 
the  Continental  U.S.  and  in  several  other  countries  around  the  world.  Figure  6  shows  the 
locations  of  all  twenty-three  AOC  sites. 


Figure  6.  Air  and  Space  Operations  Center  Location  Map.  (From  Murray,  2008). 

AOCs  are  divided  into  four  different  groups  according  to  the  type  of  operational 

support  they  provide  to  the  Joint  Force  Air  Component  Commander  (JFACC).  The  four 

groups  are  Falconer,  Tailored,  Functional  and  Support.  All  AOCs,  on  short  notice,  can 

provide  the  JFACC  with  the  appropriate  air  assets  and  trained  personnel  needed  to 

respond  to  threats  originating  from  both  inside  and  outside  the  continental  U.S.  Falconer 

AOCs  are  responsible  for  directly  supporting  their  assigned  theater  regional  commander. 

Tailored  AOCs  support  missions  ranging  from  homeland  security  to  strategic  defense. 

Functional  AOCs  are  tasked  with  supporting  the  U.S.  Strategic  Command  and  the  U.S. 
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Transportation  Command.  The  other  support  AOCs  across  the  U.S.  undertake  roles 
dealing  with  training  personnel  along  with  various  other  support  related  functions. 

The  push  to  ensure  Commanders  have  the  greatest  and  newest  technology 
available  to  them  in  order  to  protect  the  U.S.  homeland  and  our  way  of  life  is  evident  in 
the  integration  of  AOCs  across  the  country.  In  May  2007,  the  dual-hatted  Commander 
of  the  12th  AF  and  AFSOUTH  opened  the  first  and  only  U.S.  “Falconer”  AOC  at  the 
time,  at  Davis  Monthan  AFB  in  Arizona.  The  new  AOC  is  responsible  for  supporting  air 
and  space  missions  in  both  Central  and  South  America  and  in  the  Caribbean  (Jackson  & 
Broshear,  2007).  The  1st  AF,  home  based  at  Tyndall  AFB  Florida,  also  opened  a  new 
Tailored  AOC  in  June  2007.  As  a  member  of  the  Air  Component  Command,  the  1st  AF 
directs  and  controls  all  activities  for  NORAD  within  the  continental  U.S. 

3,  AOCs  Designated  as  a  Weapons  System 

The  AOC  was  designed  as  a  weapon  system  by  the  Chief  of  Staff  of  the  AF, 
General  Michael  E.  Ryan,  in  September  2000.  While  visiting  Hurlburt  Field,  Florida 
aerospace  operations  center.  General  Ryan  had  the  following  to  say: 

I  declare  the  AOC  as  an  official  weapons  system  today.  During  a  real- 
world  operation,  the  AOC  will  be  the  "eyes,  ears,  hands  and  legs  of  the 
commander.  In  each  of  our  theaters,  the  ability  of  the  air  commander  to 
execute  the  missions  he  has  depends  on  the  capability  to  have  an 
aerospace  operations  center  that  (can  be  tailored)  ...  for  the  mission  he 
needs  to  do.  We  need  a  base  lining  of  the  capabilities  in  that  weapons 
system,  just  as  we  do  in  our  capabilities  in  something  like  an  F-16.  (In  the 
F-I6),  we  have  a  crew  chief  that  knows  how  to  maintain  it  and  we  have 
pilots  that  know  how  to  fly  it.  We  have  to  have  the  same  concept  for  our 
aerospace  operations  centers  (U.S.  Department  of  Air  Force,  2000,  1). 

According  to  the  Air  Force,  the  AOC  was  designated  as  a  weapon  system  to 
ensure  standardization  in  the  way  centers  are  equipped,  employed,  and  trained  (Sirak, 
2006).  This  standardization  across  AOCs  is  expected  to  (1)  produce  seamless 
coordination  between  operators,  (2)  reduce  cost  for  equipment,  (3)  decrease  compatibility 
issues  between  equipment,  and  (4)  provide  better  personnel  training  standards. 
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B.  INTERNAL  STRUCTURE  OF  AOC  AND  RESPONSIBILITIES 

1,  Internal  AOC  Organizational  Structure 

To  be  operationally  effeetive,  all  division  leaders  and  internal  offiee  personnel 
must  stay  abreast  of  the  eommon  operating  pieture  and  eoordinate  with  other  division 
personnel  in  a  timely  manner.  Coordination  and  collaboration  are  essential  to  being  able 
to  quickly  assess  the  dynamic  situation,  develop  effective  courses  of  action,  and  assign 
appropriate  assets  to  respond  to  dynamic  events.  There  are  several  internal  divisions 
within  an  AOC  with  each  division  having  specific  responsibilities  and  an  assigned  leader 
for  coordination.  Figure  7  shows  the  internal  AOC  organizational  divisional  structure. 
During  the  Research  Event,  the  Senior  Operations  Duty  Officer  (SODO),  the  Senior  Air 
Defense  Officer  (SADO),  the  Senior  Intelligence  Duty  Officer  (SIDO)  and  other  analysts 
and  key  specialty/support  personnel  (i.e..  Marine  Liaison,  Navy  Liaison,  Battlefield 
Coordination  Detachment)  were  actively  engaged  in  the  coordination  of  assets  and 
capabilities.  Additionally,  the  above  personnel  relayed  and  inputted  critical  data  and 
information  into  the  team  macrocognition  process. 


Offensive 


Defensive 


ISR 


Speciaity/Support 


SODO 

SADO  -I 

ODOs 

DDOs 

Dynamic 

targeting 

TMD 

iCO  J 

-  Direct  tasking/direct  support 

-  Coordination  support 

Note;  This  organizational  chart  represents  a 
notional  Combat  Ops  division  for  a  major 
operation.  The  type  of  operation  may  require 
changes  in  organization. 


Ligure  7.  Air  and  Space  Operations  Center  Internal  Organization  Structure  (Lrom  Air 
Lorce  Tactics,  Techniques,  and  Procedures  2-3.2,  2004). 
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2.  Offensive  Division  and  SODO  Responsibilities 

The  Offensive  Division  is  led  by  the  Senior  Operations  Duty  Officer  (SODO)  and 
contains  several  offensive  duty  officers  and  the  DTC.  The  SODO  and  offensive  duty 
officers  are  responsible  to  the  Chief  of  Combat  Operations  for  ensuring  that  air  tasking 
orders  operations  meet  the  JFACC  objectives.  On-going  collaboration  with  the  Wing 
Operations  Center  and  other  service  control  agencies  using  the  contingency  theater 
automated  planning  system  is  necessary  to  ensure  air  assets  are  used  effectively  to  meet 
objectives.  The  SODO’s  specific  coordination  actions  include  analyzing  air  tasking 
requests  to  ensure  all  tasks  were  attainable;  tracking  all  air  tasking  order  missions  and 
developing  of  alternate  courses  of  action  in  case  of  emergency  or  change  in  the  common 
operating  picture;  and  ensuring  all  changes  and  modifications  to  the  air  tasking  order 
were  coordinated  with  appropriate  units  and  disseminated  to  all  units  affected/concerned 
(12“^  Air  Force  AFFOR,  1996). 

3.  Defensive  Division  and  SADO  Responsibilities 

The  Defensive  Division  is  controlled  by  the  Senior  Air  Defense  Officer  (SADO) 
and  contains  several  defense  duty  officers,  theater  missile  defense,  and  the  interface 
control  officer.  The  SADO  and  defense  duty  officer  are  responsible  to  the  chief  of 
combat  operations  for  all  air  defense  operations.  The  SADO  and  defense  duty  officer 
responsibilities  include  joint  planning,  coordination,  employment  and  life-cycle 
management  of  all  air  defense  systems  in  the  theater.  Additionally,  they  assist  in 
developing  courses  of  action  for  countering  enemy  offensive  air  activities  and  serve  as 
the  airspace  control  authority  developing  standard  operating  procedures  for  air  battle 
operations.  Effective  collaboration  between  the  SADO,  defensive  duty  officers,  theater 
missile  defense  and  interface  control  officer  is  considered  essential  to  ensure  protection 
of  U.S.  and  ally  forces  and  civilians  located  within  the  area  of  responsibility  (Joint 
Warfare  Publication,  2003). 

4.  ISR  Division  and  SIDO  Responsibilities 

The  Intelligence,  Surveillance,  Reconnaissance  Division  (ISRD)  is  managed  by 

the  Senior  Intelligence  Duty  Officer  (SIDO)  and  contains  the  ISR  cell,  which  employs 
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several  colleetion  managers,  analysts,  and  targeteers.  The  ISRD  is  responsible  for 
developing  the  overall  ISR  strategy  for  the  AOC  and  planning  and  exeeuting  ISR 
missions.  Additionally,  the  ISRD  performs  detailed  assessments  and  in-depth  analysis  of 
the  battle  spaee  environment  and  disseminated  the  information  to  other  Researeh  Event 
participants.  (United  States  Air  Force  Central  Command,  2008).  Figure  8  shows  a  more 
detailed  list  of  ISRD  assessment  and  planning  functions  that  must  take  place  in  every 
AOC  or  training  scenario  to  enable  the  different  internal  cells  to  complete  their  assigned 
missions. 


Figure  8.  ISRD  Assessment  and  Planning  (From  United  States  Air  Force  Central 

Command,  2008). 


5,  Specialty/Support  Division  Responsibilities 

Several  key  personnel  in  the  specialty/support  division  provide  necessary  support 
to  AOC  internal  cells  performing  live  or  training  missions,  such  as  the  JAG  Officer, 
Battlefield  Coordination  Detachment,  Naval  Aviation  Fiaison  Element  (NAFF),  Marine 
Fiaison  Officer  (MARFO),  and  the  Special  Operations  Fiaison  Element  (SOFE).  The 
JAG  Officer  provides  legal  guidance  to  the  internal  AOC  divisions  and  operational 
commanders  located  in  the  area  of  responsibility  to  assist  with  rules  of  engagement 
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determination.  The  battlefield  eoordination  detaehment  is  typically  an  Army  assigned 
liaison  officer  who  is  responsible  for  handling  and  processing  tactical  air  support  and  de¬ 
conflicting  air  operations  for  the  Army  component  in  theater.  The  battlefield 
coordination  detachment  assists  in  planning  and  coordinating  air  operations  by 
interpreting  and  introducing  Army  ground  warfare  techniques  (Joint  Publication  1-02, 
2001).  The  NALE  and  MARLO  liaisons  support  by  planning  and  coordinating  naval  and 
Marine  Corp  specific  capabilities  such  as  air  support,  ISR  capabilities,  and  amphibious 
assault  operations.  The  NALE  counsels  the  AOC  internal  division  planners  of  the 
maritime  domain  awareness  picture  and  the  MARLO  advises  the  AOC  internal  division 
on  marine  ground  specific  operations  and  techniques  (12**'  Air  Eorce  Air  Eorces  Eorce, 
1996). 


C.  COLLABORATION  BETWEEN  OPERATIONAL  PLAYERS 

1.  Time  Sensitive  Target  Contributing  Factors  and  Process 

The  AOC  is  tasked  with  carrying  out  both  static  and  dynamic  operations.  Static 
operations  include  maintaining  the  theater  common  operating  picture,  developing  and 
analyzing  ISR  data  to  maintain  accurate  situation  awareness,  supplying  weather 
forecasting  services,  and  providing  Judge  Advocate  General  legal  services  for  rules  of 
engagement  determination.  Dynamic  operations  can  include  finding  and  destroying 
SCUD  missile  sites;  locating,  tracking  and  destroying  radiological  or  explosively  formed 
projectiles  logistic  networks;  and  locating,  tracking,  and/or  exploiting  key  leadership 
targets  (Air  Force  Research  Laboratory  Warfighter  Readiness  Division,  2008).  The 
method  employed  by  AOCs,  depicted  in  Figure  9,  is  the  Find,  Fix,  Track,  Target,  Engage, 
and  Access  (F2T2EA)  process  which  enables  better  command  and  control  decision  flow 
and  sensor  coordination  (Joint  Publication  3-60,  2007). 

The  complexity  inherent  in  performing  the  tasks  involved  in  static  operations 
coupled  with  a  constantly  changing  environment  of  dynamic  mission  factors  (i.e.,  number 
of  targets,  type  of  targets,  and  speed  of  targets)  present  challenging  problems.  These 
extremely  challenging  problems  can  be  difficult  to  assess,  orchestrate  measures  and 
countermeasures,  and  implement  courses  of  action  successfully  without  having  effective 
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and  efficient  coordination  and  de-confliction  within  the  AOC  internal  organization 
divisions.  Collaboration  between  the  internal  divisions  is  paramount  to  maintaining  an 
up-to-date  situational  awareness  picture,  developing  the  correct  course  of  action  to 
respond  to  individual  or  multiple  threats,  and  assigning  the  correct  weapon  to  the  right 
mission. 


Figure  9.  Find  Fix  Track  Target  Engage  and  Asses  Dynamic  Targeting  Model  (From 

Joint  Publication  3-60,  2007). 

2,  Dynamic  Targeting  Cell  Responsibilities 

The  DTC  is  an  optional  cell  within  the  AOC  and  is  only  stood  up  under  the 
control  of  the  SODO  when  there  are  too  many  time  sensitive  targets  for  the  regular  AOC 
internal  organization  structure  to  process  in  a  timely  and  effective  manner.  The  DTC  has 
a  minimized  chain  of  command  and  a  streamlined  F2T2EA  standard  operating  procedure 
that  supports  quicker  decision-making  and  target-prosecution  times.  The  DTC  relies 
heavily  on  ISR  and  other  information  collected  from  the  other  internal  AOC  divisions  but 
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employs  a  ground  track  coordinator,  an  attack  coordinator,  a  command  and  control  duty 
officer,  and  a  target  duty  officer.  Figure  10  shows  the  DTC  structure  in  the  dashed  box. 
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Figure  10.  Dynamic  Targeting  Center  Structure  (After  Case,  Koterba,  Conrad,  Okerman, 

&  Vanderberry,  2006). 


During  the  Research  Event,  the  attack  coordinator  and  target  duty  officer  were 
responsible  for  assessing  the  situation,  selecting  an  appropriate  weapon  for  the  target 
selected,  and  planning  the  mission.  The  attack  coordinator  also  accounted  for  verifying 
that  the  TST  target  was  not  on  the  no  strike  or  no  kill  list  (Shebilski,  Freeman,  Levchuk, 
&  Gildea,  2008).  The  command  and  control  duty  officer  drafts  the  air  tasking  orders  for 
all  TST  missions.  To  assist  with  the  communication  and  collaboration  between  the 
various  DTC  and  AOC  internal  division  personnel,  the  joint  automated  deep  operations 
coordination  system  collaboration  software  application  tool  is  used  as  the  primary 
interface  between  divisions. 
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3,  Find  Fix  Track  Target  Engage  and  Assess  Cognitive  Processes 

The  Air  Force  uses  the  F2T2EA,  or  “Kill  Chain”  as  it  is  often  referred  to,  to 
conduct  dynamic  targeting  against  TSTs.  The  kill  chain  provides  AOC  decision  makers 
the  means  by  which  to  quickly  engage  targets  of  opportunity.  According  to  the  Joint 
Targeting  publication  (Joint  Publication  3-60,  2007),  TSTs  are  targets  of  very  high 
importance  to  the  Joint  Force  Commander’s  mission  and  pose  a  significant  threat  to 
coalition  forces.  Our  enemies  today  are  aware  of  our  tactics,  procedures,  and  asset 
capabilities  and  use  this  knowledge  to  plot  attacks  against  us.  The  enemy’s  ability  to 
quickly  adapt,  willingness  to  hide  in  caves  for  long  periods  of  time,  and  readiness  to  use 
civilian  populations  as  human  shields,  severely  challenge  the  AOC  commander’s 
cognitive  processes  and  ability  to  effectively  apply  the  F2T2EA  targeting  steps  to  counter 
TSTs. 


a.  Step  1  “Find” 

The  first  step  of  dynamic  targeting  is  “find.”  Based  on  prior  intelligence 
received,  AOC  commanders  allocate  ISR  assets  to  search  for  TSTs.  ISR  assets  such  as 
airborne  warning  and  control  system  aircraft,  joint  surveillance  target  attack  radar  system, 
unmanned  aerial  vehicles,  and  reconnaissance  aircraft  provide  the  AOC  with  the  most  up- 
to-date  intelligence  available.  The  ability  to  direct  and  task  the  often  limited  and  the 
various  types  of  ISR  assets,  when  and  where  they  are  needed,  requires  skill  and  is  a 
critical  challenge  for  any  commander  conducting  air  operations.  It  can  also  be  extremely 
difficult  for  those  analysts  responsible  for  analyzing  and  trying  to  decipher  the  usually 
large  amounts  of  data.  Because  ISR  assets  are  needed  to  support  all  types  of  missions, 
they  are  not  always  immediately  available  when  needed.  The  cognitive  processing  ability 
of  humans  is  limited  which  can  make  it  difficult  to  process  and  synthesis  the  extremely 
large  amount  of  data  and  information  that  is  often  received.  To  make  matters  worse,  in 
order  to  be  effective  in  a  dynamic  environment  against  TSTs,  those  processing  the 
information  must  do  so  on  a  compressed  time-line.  The  window  of  opportunity  to 
prosecute  TSTs  is  short  and  incomplete  due  to  the  style  of  warfare  the  enemy  employs 
and  the  decision  to  act  must  often  be  made  based  on  incomplete  information. 
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b.  Step  2  ‘Tix” 

Fix,  the  second  step  in  dynamic  targeting,  refers  to  the  ability  to  use 
available  ISR  assets  to  locate  TSTs  (Joint  Publication  3-60,  2007).  The  process  of 
fixing  the  target  involves  analyzing  data  by  experienced  personnel  to  provide  an  accurate 
picture  of  the  battle  space  to  support  AOC  commanders  in  assigning  specific  sensors  at 
the  correct  time  and  intervals.  This  process  can  be  a  long  and  daunting  task  because 
information  received  from  various  sensors  can  often  be  ambiguous  and  incomplete.  AOC 
commanders  must  have  the  necessary  experience  to  recognize  when  they  have  enough 
information  or  what  additional  information  they  need  to  support  their  developing  and 
achieving  accurate  situation  awareness.  This  task  can  be  a  very  stressful  endeavor  for 
those  personnel  responsible  for  asset  management  and/or  making  higher  level  decisions 
due  to  the  high  workload  produced  by  needing  to  process  large  amounts  of  information 
under  severely  time  compressed  conditions  with  high  consequences  for  failure.  The 
decision  maker  must  be  able  to  deconflict  multiple  air,  land  and  sea  assets,  and  coordinate 
and  direct  ISR  assets  to  collect  information  on  an  enemy  that  is  extremely  gifted  at 
concealing  themselves.  Juggling  the  multiple  tasks  can  produce  information  overload  for 
decision  makers  and  analysts.  Overload  consequences  can  contribute  to  poor  higher-level 
decisions  and  poor  intelligence  collection  creating  a  loss  of  SA  and  resulting  in  greater 
casualties. 


c.  Step  3  “Track” 

Track,  the  third  step  in  dynamic  targeting,  starts  after  a  fix  is  obtained. 
Assets  are  then  assigned  to  continually  observe  and  monitor  the  TST  (Joint  Publication 
3-60,  2007).  In  a  dynamic  environment,  a  TST  can  change  location  within  minutes.  To 
keep  assets  tracking  the  target,  decision  makers  and  other  personnel  involved  must 
maintain  accurate  SA  to  be  able  to  immediately  react  to  sudden  changes.  Maintaining  SA 
requires  the  decision  maker  to  keep  track  of  multiple  processes  and  events  which  can  tax 
the  operator’s  working  memory  and  creates  a  high  cognitive  workload.  To  make  the 
right  decisions,  the  decision  maker  needs  to  constantly  share  information  and  build  new 
knowledge  to  stay  abreast  of  the  evolving  situation.  New  knowledge  is  accomplished 
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through  collaboration  with  other  internal  and  external  team  members  who  are  analyzing 
various  data  sets  reeeived  from  various  sensors  and  then  organizing  the  data  into  a 
meaningful  strueture.  Dynamie  interactions  between  personnel  responsible  for  colleetion 
and  those  responsible  for  information  proeessing  must  be  eontinuous  to  ensure  the  correet 
sensors  are  assigned  to  the  right  type  of  targets  at  the  preeise  time.  AOC  deeision  makers 
must  have  the  ability  to  proeess  a  high  volume  of  information  under  time-eompressed 
eonditions  and  then  make  time-eritieal  responses  to  various  situations  in  stressful 
environments  (high  workload,  eomplex  soeio-teohnieal  environment  and  high 
eonsequenees  for  errors). 

d.  Steps  4  and  5  “Target  and  Engage” 

The  fourth  and  fifth  steps  in  the  kill  ehain  are  target  and  engage.  At  this 
point,  the  AOC  eommander  has  the  neeessary  information  to  engage  and  take  appropriate 
aetions  to  eause  a  desired  effeet  (Joint  Publieation  3-60,  2007).  Even  with  preeise 
targeting  information,  other  eontributing  faetors,  if  not  properly  handled,  ean  eause 
serious  delays  and  jeopardize  the  mission.  The  final  decision  regarding  whether  or  not  to 
engage  a  target  may  also  depend  on  the  ability  to  de-eonfliet  assets  in  the  surrounding 
area  to  prevent  “blue-on-blue”  easualties,  determination  of  rule  of  engagement 
authorizing  type  of  attaek,  and  assessment  of  risk  of  eollateral  damage  to  non-military 
personnel  versus  target  value.  At  the  eonelusion  of  Operation  Desert  Storm,  25.6  pereent 
of  those  killed  were  due  to  friendly  fire  (Krepinevieh,  2003).  Deeision  makers  must  be 
able  to  assess  the  impact  of  several  additional  factors  on  their  deeision  before  giving  the 
order  or  permission  to  prosecute  a  TST. 

e.  Step  6  “Assess” 

Assess  is  the  sixth  and  final  step  in  the  kill  ehain.  To  be  effeetive  at  this 
stage,  the  eommander  must  base  his/her  decision  on  the  following:  how  relevant  and 
timely  is  the  intelligenee  eollected;  are  the  right  sensors  being  applied  to  the  right  target 
and  at  the  right  time;  how  effeetive  are  the  weapons  being  employed  against  the  TST; 
and  is  eollateral  damage  being  kept  to  a  minimum?  To  answer  these  questions,  the 
eommander  must  be  able  to  direet  ISR  assets  and  alloeate  the  eorreet  sensors  to  eonduet 
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battle  damage  assessment.  Overhead  sensors  from  tactical  and  national  assets  provide 
imagery  and  other  intelligence  to  help  determine  the  overall  state  of  TST  targets  after 
engagement.  Based  on  the  battle  damage  assessment,  the  AOC  commander  may  decide 
that  the  target  needs  to  be  placed  on  the  targeting  list  again  or  remove  it  completely  from 
the  TST  prosecute  list.  In  a  dynamic  environment,  where  the  information  can  be 
incomplete,  missing,  or  overwhelming,  challenges  to  the  human  cognitive  processing 
ability  arise  for  the  decision  makers  who  at  a  moment’s  notice  must  be  able  to  effectively 
apply  the  kill  chain  against  a  TST  (Joint  Publication  3-60,  2007). 

D,  TEAM  COLLABORATION  MODEL 

1,  Previous  Research 

Several  previous  theses  have  conducted  research  to  evaluate  earlier  versions  of 
the  ONR  developed  model  of  team  collaboration.  Some  of  the  first  field  research 
experiments  regarding  model  evaluation  focused  on  analyzing  maritime  interdiction 
operations  (MIO)  chat  logs  from  three  MIO  exercises  and  air  warfare  audio  transcripts 
from  four  different  teams.  A  MIO  is  an  operation  that  attempts  to  delay,  disrupt  or 
destroy  an  enemy’s  supply  resources  before  they  can  be  used  to  harm  people  or  cause 
other  severe  damage  (Joint  Publication  3-03,  2007).  The  air  warfare  scenarios  involved 
team  collaboration  within  a  shipboard  Combat  Information  Center  to  identify  hostile  and 
friendly  air  contacts  (Hutchins,  Kendall,  Bordetsky,  Bourakov,  2006). 

The  first  structural  model  of  team  collaboration  validation  thesis  was  done  by 
Ensign  Maura  Garrity.  The  thesis  involved  analyzing  the  Fire  Department  of  New  York 
communication  transcript  that  recorded  several  district  and  regional  New  York  firefighter 
responses  and  actions  in  their  response  to  the  September  11,  2001,  attacks  on  the  World 
Trade  Center. 

A  second  model  of  team  collaboration  thesis  was  conducted  by  Lieutenant 
Commander  Catherine  Donaldson  and  Lieutenant  David  Johnson.  The  thesis  involved 
analyzing  a  transcript  of  recorded  audio  from  the  command  and  control  center  at 
NORAD  North  East  Air  Defense  Sector  (NEADS)  that  recorded  communications 
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between  NEADS,  the  Federal  Aviation  Administration  and  other  ageneies  during  the 
events  surrounding  the  discovery  of  the  four  hijacked  aircraft  on  September  11,  2001. 

A  third  structural  model  of  team  collaboration  thesis  was  conducted  by  Lieutenant 
Luis  Socias.  The  thesis  analyzed  a  second  communication  channel  between  NEADS, 
Federal  Aviation  Administration  and  other  agencies  during  their  response  to  the 
discovery  of  four  hijacked  aircraft  on  September  11,  2001. 
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III.  LITERATURE  REVIEW 


A.  COGNITIVE  PROCESSES 

The  word  cognition  is  derived  from  the  Latin  word  cognoscere,  which  means 
“come  to  know”  (Merriam- Webster  Dictionary,  2009).  An  individual  usually  comes  to 
know  by  developing  and  using  his  or  her  mental  processes  such  as  thinking,  knowing  and 
recollection  to  retain  intellectual  compilation  and  gain  understanding.  Cognition  is  a 
process  that  mainly  occurs  inside  the  human  brain.  The  human  brain  is  akin  to  a  central 
processing  unit  of  a  computer.  It  is  instrumental  in  taking  in  information  from  various 
sources,  processing  that  information,  and  then  storing  the  information  in  short-term  and 
long-term  memory. 

In  addition  to  the  basic  functions  of  the  brain,  the  human  brain  has  four  primary 
areas:  working  memory,  attention  and  performance,  visual  spatial  thinking,  and  learning 
recall  and  long  term  memory  (Schifferstein  &  Hekkert,  2007).  These  primary  areas  are 
crucial  to  an  individual’s  ability  to  “come  to  know.”  Gaining  understanding  by  an 
individual  or  a  group  of  individuals  who  are  functioning  together  as  a  team  is  critical  to 
the  individual  or  team’s  ability  to  effectively  perform  and  solve  complex,  dynamic 
problems.  Types  of  cognition  include  metacognition,  macrocognition  and 

microcognition. 

1.  Metacognitive  Processes 

Metacognition  can  be  defined  as  an  individual’s  own  understanding  of  his  or  her 

cognitive  processes  and  that  person’s  ability  to  effectively  use  that  unique  understanding 

as  knowledge  to  improve  their  cognitive  processes  (Schifferstein  &  Hekkert,  2007).  In  a 

nutshell,  metacognition  can  be  referred  to  as  “thinking  about  thinking”.  An  example  of 

metacognition  would  be  that  an  individual  notices  that  he  or  she  is  having  a  hard  time 

learning  event  A  compared  to  event  B  and,  therefore,  needs  to  double  check  event  A  to 

ensure  comprehension.  There  are  three  classes  of  metacognition:  (1)  knowledge  (i.e., 

what  an  individual  knows  about  their  own  cognitive  processes);  (2)  regulation  (i.e.,  an 
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individual’s  own  internal  mechanisms  used  to  control  their  cognitive  learning);  and  (3) 
experiences  (i.e.,  an  individual’s  own  experiences  with  the  current  cognitive  task) 
(Flavell,  1979). 

Individuals  who  have  developed  and  can  exercise  sound  metacognitive  techniques 
are  able  to  control  and  focus  their  learning  processes.  This  can  drastically  improve  their 
ability  to  work  more  effectively  and  perform  increasingly  more  dynamic  tasks. 
Metacognition  is  usually  measured  and  evaluated  in  a  laboratory  environment  by 
administering  a  pre-test  before  the  event  to  test  pre-existing  knowledge.  Then  some  of 
the  individuals  are  shown  metacognition  techniques  (Education  Resources  Information 
Center  Digest,  1990).  After  the  techniques  are  reviewed,  all  individuals  start 
participating  in  the  event  or  scenario.  After  the  event  is  over,  post-tests  are  administered 
to  all  participants  and  the  results  between  pre-test  and  post-test  are  analyzed  to  determine 
if  metacognition  techniques  improved  the  individual’s  knowledge  level  and/or  over  all 
event  performance. 

2,  Macrocognitive  Processes 

Macrocognition  is  defined,  for  this  research,  as  the  internalized  and  externalized 
high-level  mental  processes  employed  by  teams  to  create  new  knowledge  during 
complex,  one-of-a  kind,  collaborative  problem  solving  (Letsky,  Warner,  Fiore,  Rosen,  & 
Salas,  2007).  High-level  mental  processes  refer  to  the  cognitive  processes  involved  in 
combining,  visualizing,  and  aggregating  information  to  resolve  ambiguity  in  support  of 
the  discovery  of  new  knowledge  and  relationships.  Macrocognition  studies  concentrate 
on  assessing  complex  problems  of  the  naturalistic  real  world  in  contrast  to 
microcognition  studies  which  are  typically  focused  and  limited  to  laboratory  type 
experiments.  Some  of  the  complex  problems  macrocognition  studies  include  making  key 
operational  decisions  that  could  affect  the  lives  of  individuals  under  poorly  defined 
situations  and  under  time  pressures.  Macrocognition  focuses  on  both  internal  and 
external  processes  and  how  they  affect  the  decision  maker.  Figure  11  shows 
macrocognitive  functions  and  supporting  processes  that  would  be  used  by  Dynamic 
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Targeting  Cell  (DTC)  personnel  in  executing  the  Find  Fix  Track  Target  Engage  and 
Asses  (F2T2EA)  process  used  while  prosecuting  time  sensitive  targets  (TSTs). 


Eigure  1 1 .  Macrocognitive  and  supporting  processes  for  individuals,  teams,  and 
information  technologies  (Erom  Klein,  Ross,  Moon,  Klein,  Hoffman,  & 

Hollnagel,  2003). 


3.  Microcognitive  Processes 

Microcognition  is  recognized  as  including  the  essential  stepping  stones  for  which 
thinking  and  processing  are  calculated  and  performed.  Memory  associated  with 
microcognitive  processes  allow  individual’s  to  cope  with  more  dynamic  situations  and 
handle  a  greater  complexity  of  information  processing.  Microcognition’s  primary  focus 
is  to  try  to  understand  how  the  invisible  processes  that  transpire  inside  an  individual’s 
brain  such  as  reasoning  aptitude  and  mental  processing  capability  occur  and  how  they  are 
directly  affected  either  favorably  or  adversely  (Klein,  Ross,  Moon,  Klein,  Hoffman,  & 
Hollnagel,  2003).  Some  microcognition  experts  conduct  experiments  looking  for 
communication  design  or  team  organizational  model  issues  that  can  negatively  impact  an 
individual  or  team’s  ability  to  complete  assigned  tasks.  Poor  designs  can  make  it  harder 
on  individuals  to  retrieve  important  task  relevant  knowledge  (i.e.,  knowledge  bottleneck); 

remember  or  store  into  memory  specific  key  task  related  details  (i.e.,  memory 
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bottleneck);  and/or  make  it  more  difficult  to  stay  focused  on  the  task  at  hand  or  switch 
back  and  forth  from  different  tasks  (i.e.,  attention  bottleneck)  (Schifferstein  and  Hekkert, 
2007).  Figure  12  shows  the  situational  design  interface  representing  cognitive  processes 
involved.  In  order  to  assess  microcognition,  eye  tracking  sensors  and  other  measurement 
devices  must  be  properly  designed,  implemented,  and  assessed.  The  study  of 
microcognition  is  typically  done  in  a  laboratory  environment  with  well  defined  attributes 
and  goals. 


LonK-ti-rm  Mrmnry 

Knonlrdff  and  Skilh 

(menfij!  tttadeii.  procfiftire^.  itrate^iest 


Figure  12.  Situational  Design  and  Cognitive  Model  Process  (From  U.S.  Department  of 

Transportation,  2009). 

B,  TEAM  PERFORMANCE 
1,  Performance  Factors 

Performance  is  defined  as  an  individual’s  ability  to  execute  one  or  several  tasks 
with  a  specific  end  result  (Merriam- Webster  Dictionary,  2009).  There  are  several 
internal  and  external  factors  that  affect  an  individual  or  a  group’s  ability  to  carry  out 
specific  tasks  in  support  of  unified  missions.  Internal  factors  that  affect  performance 
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include  individual  feelings,  emotions,  skill  level,  and  inherent  core  values.  External 
factors  include  the  organizational  structure  hierarchy  (i.e.,  distributed  team,  centralized 
team,  etc...),  and  situational  environment  factors  (i.e.,  time,  space,  force  and  other 
dynamic  elements).  Another  key  factor  that  can  affect  performance  is  the  common 
understood  situational  awareness  between  individual  teams  and  team  members.  In  a 
highly  dynamic  environment  supporting  multiple  theater  operations,  it  is  very  easy  for  an 
individual  to  become  inundated  with  too  much  information  leading  to  information 
overload  resulting  in  poor  decision  making.  Figure  13  is  a  performance  topology  map 
that  shows  the  relationship  between  both  internal  and  external  factors  that  affect 
performance. 


Figure  13.  Performance  Topology  Map  (From  Clark,  2004). 


31 


2. 


Team  Performance  Metrics  and  Improvements 


Performance  metrics  can  be  established  by  identifying  all  the  critical  processes  for 
accomplishing  the  individual  or  team  mission  objective(s).  Next  standard  operating 
procedures  are  developed  that  are  used  to  accomplish  ah  the  identified  critical  processes. 
Finally,  a  value  is  assigned  to  each  accomplished  process  based  off  some  variable  related 
to  the  task  (i.e.,  time  of  completion,  accuracy  rating,  and  communication  percentage). 
These  values  can  then  be  used  as  a  baseline  score  that  can  be  compared  to  other  changes 
made  to  the  system  in  order  to  determine  if  changes  in  the  organizational  structure, 
processing  procedures,  or  human  interaction  resulted  in  positive  or  negative  results. 

Overall  performance  can  be  drastically  improved  by  ensuring  that  ah  individuals 
understand  the  operational  mission  objectives  and  performance  goals.  Each  member 
should  be  briefed  on  common  factors  to  ensure  individual  mental  models  are  consistent 
with  one  another  to  ensure  everyone  interprets  the  goals  with  the  same  mind  set. 
Additionally,  continued  metric  studies  of  the  organizational  hierarchy  and  standard 
operating  procedures  should  be  done  on  a  regular  basis  to  make  certain  the  best  structural 
model  and  procedures  are  being  used  to  accomplished  goals  (Nikols,  2003). 

C.  DECISION  MAKING 

1.  Decision-making  Methods 

Research  in  decision  making  is  a  long  and  ongoing  process  that  continues  to  look 
at  how  people  make  decisions.  There  are  various  methods  and  combinations  of  decision¬ 
making  techniques  that  affect  how  decisions  made.  The  traditional  method  of  decision 
making  continues  to  be  one  of  the  preferred  methods  consistently  used  in  both  civilian 
and  military  environments.  According  to  Orasanu  and  Connolly,  traditional  decision 
makers  systematically  go  through  ah  possible  alternatives  before  choosing  the  course  of 
action  that  offers  the  best  outcome.  This  method  for  making  decisions  allows  the 
decision  maker  to  choose  their  final  choice  based  on  known  goals,  purposes,  and  values 
(Klein,  Orasanu,  Calderwood,  &  Zsambok,  1993).  A  more  recent  decision-making 
method  has  been  described  by  Dr.  Gary  Klein  called  recognition-primed  decision 
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making.  Klein  believes  that  people  make  deeisions  mueh  faster  and  without  eomparing 
the  choiees,  as  shown  in  studies  of  both  naturalistie  deeision  making  and  reeognition- 
primed  deeision  making. 

2.  Classical  Approach 

Classical  decision  making  is  a  cognitive  progress  by  which  personnel  in  decision 
roles  follow  a  step-by-step  procedural  and/or  repeatable  way  of  coming  up  with  the  best 
choice  for  a  given  situation.  Decision  makers  arrive  at  what  they  feel  is  the  best  solution 
by  logically  going  through  a  list  of  choices  and  weighing  the  pros  and  cons  of  each  option 
and  decision  before  settling  on  a  final  choice.  The  process  involves:  identifying  a  set  of 
rules  and  boundaries  of  the  problem;  establishing  a  procedure  or  system  to  evaluate  all 
options;  weighting  each  option  outcome  against  all  other  option  outcomes;  and  finally, 
performing  a  grading  of  the  options.  The  decision  maker  will  pick  the  option  that 
provides  them  with  the  highest  or  lowest  score  depending  on  how  grading  rules  were 
setup  (Klein,  1999). 

According  to  Beach  and  Lipshitz,  many  decision  makers  resist  this  classical 
decision-making  method  because  it  is  cumbersome  and  takes  lots  of  the  decision  maker’s 
time  (Klein,  et  ah,  1993).  In  an  experiment  conducted  at  Massachusetts  Institute  of 
Technology  Sloan  School  of  Management,  students  in  their  final  year  of  college  were 
studied  to  see  what  decision  strategies  they  would  use  to  select  their  future  job. 
Surprisingly,  students  choose  to  go  with  their  gut  feelings  rather  than  sequentially  going 
through  the  steps  to  come  up  with  the  best  choice  (Klein,  1999).  A  similar  observation  of 
firefighter  commanders,  by  Klein,  operating  in  stressful  environments  concluded  that 
firefighters  with  over  twenty  years  of  experience  behaved  in  the  same  manner  as  students 
in  the  MIT  experiment  (Klein,  1999). 

Although  decision  makers  may  use  the  traditional  systematic  method  in  which 
traditional  decision  making  is  performed,  classical  decision  making  remains  a  method 
that  is  widely  used  and  is  extremely  useful  in  various  situations.  In  a  strategic 
environment,  where  traditionally  time  and  speed  are  not  critical  factors,  the  classical 
approach  allows  the  decision  makers  to  take  a  step  back,  look  at  all  the  different 
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operational  and  tactical  players  involved,  consider  the  various  military  and  political 
options  and  implications,  weigh  the  choices  against  one  another,  and  finally  pick  the  best 
choice  that  would  minimize  casualties  and  guarantee  mission  success.  Classical  decision 
making  also  works  well  at  commands  that  have  primary  missions  of  dictating  and  writing 
policies. 

In  a  dynamic  environment,  such  as  an  AOC  conducting  missions  over  Iraq  and 
Afghanistan,  the  classical  approach  will  not  work.  AOC  commanders  do  not  have  the 
time  to  fully  run  through  all  the  different  possible  options  before  deciding  on  a  course  of 
action.  The  commander’s  window  of  opportunity  to  engage  high-value  targets  could  last 
for  only  a  few  minutes  or  even  only  seconds  which  means  the  commander  must  have  the 
necessary  training  and  experience  to  rapidly  decide  what  are  the  best  techniques  and 
assets  to  deploy  for  a  given  target.  Klein  referred  to  decisions  made  in  a  time  limited 
environment  naturalistic  decision  making. 

3.  Naturalistic  Approach 

Naturalistic  decision-making  is  the  “study  of  how  people  use  their  experience  to 
make  decisions  in  field  settings”  (Klein,  1999).  Unlike  classical  decision  making,  where 
decision  makers  follow  an  orderly  process  before  finalizing  their  decision,  during 
naturalistic  decision  making,  decision  makers  access  past  experience  that  applies  to  the 
current  situation.  According  to  Klein,  time  pressure,  high  stakes,  experienced  decision 
makers,  inadequate  information,  ill-defined  goals,  poorly  defined  procedures,  cue 
learning,  context,  dynamic  conditions,  and  team  coordination  are  all  features  that  set 
naturalistic  decision  making  apart  from  classical  decision  making  (Klein,  1999).  Human 
decision  making  research  conducted  by  Orasanu  and  Connolly  leads  us  to  believe  that  the 
above  decision  making  features  (i.e.,  time  pressure)  have  been  left  out  of  past  decision 
making  research  resulting  in  an  incomplete  view  into  the  human  decision  making  process 
(Klein,  et  ah,  1993). 

An  AOC  commander  conducting  time-sensitive  targeting  faces  some  if  not  all  of 
the  above  naturalistic  decision-making  features.  The  commander,  based  on  his  or  her 
level  of  experience,  must  quickly  decide  on  asset  allocation  and  weapons  employment  for 
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a  given  target.  Klein  estimated  that  firefighter  commanders  make  90  percent  of  their 
decisions  in  less  than  one  minute  (Klein,  1999).  The  experience  level  of  the  commander 
plays  a  major  role  in  their  decision  making  capabilities.  Additionally,  past  mission  goals 
and  priorities,  rules  of  engagement  limits,  reliable  intelligence,  and  predictable  casualty 
rates  are  other  features  of  importance  to  the  commander.  These  other  features  are  stored 
within  an  experienced  commander’s  memory  and  are  quickly  recognized  and  used  in  the 
commander’s  decision-making  process.  Klein  refers  to  this  as  recognition-primed 
decision  making. 

4,  Recognition-Primed  Decision 

Klein,  while  working  with  firefighter  commanders,  noticed  that  experienced 
firefighters  in  a  given  situation  will  immediately  recognize  the  most  critical  factors  and 
then  decide  and  implement  the  best  course  of  action.  The  past  experience  of  the 
commanders  immediately  takes  over  and  without  having  to  go  through  a  list  or 
comparing  possible  options,  the  commander  is  able  to  decide  and  act.  To  an  experienced 
decision  maker,  the  classical  way  of  coming  up  with  a  decision  wastes  valuable  time, 
which  may  result  in  more  lives  lost  and  could  be  the  difference  between  winning  or 
losing  the  fight.  Klein  believes  that  recognition  prime  decision  making  is  a  more 
strategic  and  clever  way  of  using  one’s  own  experience  to  quickly  come  to  a  decision 
(Klein,  1999). 

Recognition  prime  decision  making  is  the  fusion  of  a  decision  maker’s  use  of 
experience  to  size  up  a  given  situation  in  order  to  come  up  with  a  course  of  action  and  the 
manner  in  which  he  or  she  mentally  simulates  implementation  of  the  course  of  action. 
The  observations  and  interviewing  of  experienced  firefighters  and  military  decision 
makers,  under  time  stressed  environments,  provided  Klein  with  conclusive  data  on  how 
people  use  experience  to  make  decision.  Klein  believes  that  a  decision  maker’s 
experience  determines  his  or  her  course  of  action  and  such  a  decision  is  normally  made 
without  the  use  of  classical  decision  making  techniques  (Klein,  1999).  Figure  14  is 
Klein’s  model  on  recognition  prime  decision  making.  The  model  shows  how  an 
experienced  decision  maker  mentally  steps  through  developing  a  single  course  of  action. 
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Variation  3 

Evaluate  Courae  ot  Action 


Experience  the  Situation  in  a  Cnanging  Context 


Perceived  as  typical 
(Prototype  or  Analogue] 


Recognition  tias  tour  by-products 


Evaluate  ^ 
Action  <n) 

[Mental  Simulation] 


C^Mt^iTy  ^  ^  C  Will  it  wortc'^] 
yes.  but 


no 


yes 


Implement 
Course  of  Action 


Figure  14.  Recognition  Prime  Decision  Model  (From  Klein,  1993). 
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IV.  MEASUREMENT  MODEL  FOR  MACROCOGNITION 

RESEARCH 


A.  MODEL  MACROCOGNITION  FOCUS 


The  concept  of  macrocognition  is  to  develop  a  better  understanding  of  the 
cognitive  processes  involved  when  a  team  collaborates  to  solve  a  unique,  complex,  time- 
compressed  problem  (Letsky  et  ah,  2007).  Figure  15  is  the  team  measurement  model  for 
macrocognition  research  that  shows  the  relationship  between  different  phases  of 
knowledge  building  and  developing  understanding  during  team  collaboration.  This 
measurement  model  was  developed  to  try  and  capture  and  measure  the  macrocognitive 
processes.  The  term  macrocognition  was  coined  to  capture  and  distinguish  higher-level 
cognitive  processes  used  by  individuals  and  teams  from  lower-level,  or  microcognitive 
processes  (e.g.  attention,  perception,  and  memory)  (Letsky  et  ah,  2007). 


Legend: 


Outputs) 


Figure  15.  Team  Measurement  Model  for  Macrocognition  Research  (From  Fiore  et  ah,  in 

press) 
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To  better  understand  the  relationship  between  the  different  phases  of  the 
measurement  model,  shown  in  Figure  15,  one  must  understand  that  there  is  not  initial 
model  starting  point.  Different  team  dynamics  (i.e.,  personnel,  asynchronous  team)  and 
assigned  tasks  will  dictate  what  phase  of  the  model  is  used.  The  focus  of  macrocognition 
is  on  building  new  knowledge  in  real-word  settings  when  a  team  collaborates.  Figure  16 
illustrates  how  unstructured  raw  pieces  of  datum  are  transformed  into  integrated 
actionable  knowledge. 


Figure  16.  Knowledge  Building  Process  in  Macrocognition  (From  Fiore  et  ah,  in  press) 

Macrocognition  is  demonstrated  during  the  conversion  of  internalized  team 
knowledge  into  externalized  team  knowledge  through  the  individual  knowledge  and  team 
knowledge  building  processes  (Fiore,  et  ah,  in  press,).  Data  is  considered  to  be  raw  and 
unprocessed  bits  of  particulars.  For  data  to  be  transformed  into  information,  it  must  be 
organized  and  referenced  in  some  context  thus  being  considered  processed  information. 
Information  is  considered  to  be  knowledge  when  it  is  organized  in  such  a  way  that  allows 
it  to  be  understood  and  used  to  solve  a  problem  or  direct  actions.  Table  2  provides  a 
formal  definition,  example  and  explanation  of  data,  information  and  knowledge. 
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Table  2. 


Data,  Information,  and  Knowledge  Definitions  (From  Fiore,  et  al.,  in 

press). 


Concept 

Explanation 

Data 

Definition 

Data  are  disparate  statements  or  faets  presented  or  represented 
separately  and  without  context. 

Example 

“The  CH-53  Marine  Corps  helieopter  ean  hoist  250  feet  with  a  600 
pound  lift  eapaeity.” 

Explanation 

Here  the  eontent  is  devoid  of  eontext  and  not  organized  in  any  way;  as 
sueh,  it  is  eonsidered  only  data. 

Information 

Definition 

Information  is  organized  or  struetured  data  (i.e.,  organized  or 
stmetured  statements  or  faets)  that  have  been  related  to  the  problem 
solving  eontext. 

Example 

“The  CH-53  Marine  Corps  helieopter  ean  earry  supplies  to  the  Red 
Cross  workers  who  have  been  taken  hostage,”  or,  “Our  three  air 
vehieles  are  the  CH-53,  the  F-16,  and  the  E2-C.” 

Explanation 

The  first  example  represents  a  transformation  in  that  it  involved 
eonneeting  the  pieee  of  data  to  the  problem.  The  seeond  example 
represents  a  transformation  in  that  it  involved  organizing  the  data  via 
eategorization  sueh  that  it  ean  serve  the  problem  solving;  resourees 
were  organized  into  eategories  of  resourees  that  serve  the  problem 
solving. 

Knowledge 

Definition 

Knowledge  is  the  integration  of  eontent  from  two  or  more  eategories  of 
information  into  something  whieh  did  not  explieitly  exist  before  and 
whieh  has  been  made  aetionable  by  being  related  to  the  problem 
solving  eontext. 

Example 

“The  CH-53  Marine  Corps  helieopter  eannot  be  used  to  earry  supplies 
beeause  it  is  foggy  over  the  southwest  eorridor  of  Nandor.” 

Explanation 

This  represents  a  transformation  beeause  vehiele  information  (a 
eategory),  was  integrated  with  weather  information  (another  eategory) 
in  sueh  a  way  that  it  serves  the  problem  solving;  that  is,  it  was  made 
aetionable  by  explaining  when  it  eould  get  (or  not  get)  supplies  to  the 
hostages. 

B,  MACROCOGNITIVE  PROCESS  DEFINITIONS 

The  foeus  of  the  maerocognitive  proeess  model  and  definitions  is  to  measure  all 
maeroeognitive  proeesses  that  oeeur  during  a  eollaborative  team  problem  solving  task. 
The  measurement  model  attempts  to  understand  the  related  proeesses  between 
internalized  knowledge,  individual  knowledge  building  proeesses,  team  knowledge 
building  proeesses  and  externalized  team  knowledge  maeroeognitive  phases 
demonstrated  during  team  eollaboration.  Data  on  internalized  knowledge  is  measureable 
by  using  eye  traeking  equipment  and  eondueting  ealeulations  on  eye  gaze.  Individual 
knowledge  building  data  is  measurable  by  eolleeting  observation  information.  Data  on 
team  knowledge  building  proeesses  is  eollected  through  eommunieations  and  non-verbal 
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communication  gestures  (i.e.,  faeial  expressions).  Externalized  team  knowledge  is 
eolleeted  through  eommunieations  and  aetual  objeets  that  are  ereated  by  the  team  (i.e., 
maps,  eharts,  and  graphs)  (Fiore  et  ah,  in  press).  The  inereased  pereeption  and 
understanding  eould  one  day  allow  researehers  to  be  able  to  aeeurately  prediet  the 
individual  and/or  group  of  teams,  generated  problem  solving  outeomes. 

C.  STAGES  OF  THE  MEASUREMENT  MODEL 

The  team  measurement  model  was  developed  from  various  iterations  of  other 
previous  maeroeognitive  models.  The  model  foeuses  on  measuring  the  maeroeognitive 
proeesses  engaged  in  by  the  team  as  they  build  knowledge  at  the  individual  and  team 
levels  (Fiore,  et  ah,  in  press).  Team  eollaboration  within  the  measurement  model  eonsists 
of  five  maeroeognitive  stages  that  show  how  members  eollaborate  with  eaeh  other  and 
with  the  team  as  a  whole.  In  eertain  stages  of  the  model,  information  ean  flow  back  and 
forth  between  eaeh  other  or  skip  stages  all  together.  There  is  no  offieial  maeroeognitive 
building  bloek  or  individual  or  team  information  proeessing  starting  point  for  the  team 
measurement  model  stages  as  theoretieally  a  team  member  ean  start  in  any  model  stage. 
For  the  purpose  of  deseribing  the  model  stages  and  their  interaetions,  in  this  thesis  we 
will  start  by  deseribing  the  flow  of  individual  and/or  team  information  proeesses  at  the 
internalized  team  knowledge  stage  flowing  into  the  individual  knowledge  building  stage 
unto  team  knowledge  building,  externalized  team  knowledge,  and  team  problem  solving 
outeomes.  Further  explanation  of  eaeh  eognitive  stage  and  their  assoeiated  eognitive 
proeessing  eodes  are  defined  in  Table  3. 
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Table  3. 


Macrocognitive  Stages  and  Assoeiated  Proeesses  included  in  the 
Measurement  Model  (From  Fiore  et  ah,  in  press). 


1 

stage  I:  Internalized  Team  Knowledge  Process:  Refers  to  the  collective  knowledge  held  in  the  individual 
minds  of  team  members.  Internalized  Team  Knowledge  is  measured  by  eliciting  it  from  individual  team 
members  using  methods  such  as  card  sorting,  concept  mapping,  paired  comparison  ratings,  scenario  probes. 

a. 

Team  Knowledge  Similarity:  Team  knowledge  similarity  can  involve  the  degree  to  which  differing  roles 
understand  one  another  (e.g.,  how  well  a  land/sea  vehicle  specialist  understands  a  humanitarian  specialist),  or 
how  well  the  team  members’  understand  the  critical  goals  and  locations  of  important  resources  (shared  situation 
awareness). 

b. 

Team  Knowledge  Resources:  Team  members’  collective  understanding  of  responsibilities  and  resources 
associated  with  the  task. 

2 

Stage  II:  Individual  Knowledge  Building  Process:  is  a  nrocess  which  includes  actions  taken  bv  individuals 
in  order  to  build  their  own  knowledge.  These  processes  can  take  place  inside  the  head  (e.g.,  reading,  mentally 
rotating  objects)  or  may  involve  overt  actions  (e.g.,  accessing  a  screenshot). 

a. 

Individual  Information  Gathering:  Individual  information  gathering  involves  actions  individuals  engage  in  to 
add  to  their  existing  knowledge  such  as  reading,  asking  questions,  accessing  displays,  etc. 

b. 

Individual  Information  Synthesis:  Individual  information  synthesis  involves  comparing  relationships  among 
information,  context,  and  artifacts  to  develop  actionable  knowledge. 

c. 

Knowledge  Object  Development:  Knowledge  object  development  involves  creation  of  cognitive  artifacts  that 
represent  actionable  knowledge  for  the  task. 

3 

Stage  III:  Team  Knowledge  Building  Process:  is  a  nrocess  which  includes  actions  taken  bv  teammates  to 
disseminate  information  and  to  transform  that  information  into  actionable  knowledge  for  team  members. 

a. 

Team  Information  Exchange:  Team  information  exchange  involves  passing  relevant  information  to  the 
appropriate  teammates  at  the  appropriate  times. 

b. 

Team  Knowledge  Sharing:  Team  knowledge  sharing  involves  explanations  and  interpretations  shared  between 
team  members  or  with  the  team  as  a  whole. 

c. 

Team  solution  Option  Generation:  Team  solution  option  generation  describes  offering  potential  solutions  to  a 
problem. 

d. 

Team  Evaluation  and  Negotiation  of  Alternatives:  Team  evaluation  and  negotiation  of  alternatives  describes 
clarifying  and  discussing  the  pros  and  cons  of  potential  solution  options. 

4 

Stage  IV:  Externalized  Team  Knowledge  Process:  Refers  to  facts,  relationshins.  and  concents  that  have  been 
explicitly  agreed  upon,  or  not  openly  challenged  or  disagreed  upon,  by  factions  of  the  team. 

a. 

Externalized  Cue-strategy  Associations:  Externalized  cue-strategy  associations  describe  the  team’s  collective 
agreement  as  to  their  task  strategies  and  the  situational  cues  that  modify  those  strategies  (and  how). 

b. 

Pattern  Recognition  and  Trend  Analysis:  Pattern  recognition  and  trend  analysis  is  the  accuracy  of  the  patterns 
or  trends  explicitly  noted  by  members  of  a  team  that  is  either  agreed  upon  or  unchallenged  by  other  team 
members. 

c. 

Uncertainty  Resolution:  Uncertainty  resolution  is  the  degree  to  which  a  team  has  collectively  agreed  upon  the 
status  of  problem  variables  (e.g.,  hostile/friendly). 

5 

Stage  V:  Team  Problem  Solving  Outcomes:  Are  assessments  of  aualitv  relating  to  a  team’s  nrohlem  solutions 
or  plan. 

a. 

Quality  of  Plan:  Quality  of  plan  (problem  solving  solution)  involves  the  degree  to  which  the  solution  adopted 
by  a  problem  solving  team  achieves  a  resolution  to  the  problem  (e.g.,  limit  fatalities,  limit  destruction). 

b. 

Efficiency  of  Planning  Process:  Efficiency  of  planning  process  describes  the  amount  of  time  it  takes  a 
problem  solving  team  to  arrive  at  a  successful  resolution  to  a  problem. 

c. 

Efficiency  of  Plan  Execution:  Efficiency  of  plan  execution  describes  the  quality  of  the  plan  (e.g.,  number  of 
lives  saved)  divided  by  the  amount  of  resources  used  to  accomplish  this  and  the  amount  of  time  the  plan  takes  to 
unfold. 
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1.  Internalized  Team  Knowledge 

The  first  stage  of  the  measurement  model  for  maerocognition  researeh  focuses  on 
Internalized  Team  Knowledge  (ITK)  which  refers  to  the  overall  combined  knowledge  of 
each  team  member  (Fiore  et  ah,  in  press).  ITK  is  collected  on  individuals  by  conducting 
various  assessments  such  as  administrating  tests  and  surveys  to  gather  past  individual 
experience.  The  knowledge  collected  on  individuals  provides  data  on  the  level  of  team 
knowledge  at  different  points  in  time  which  provides  insight  into  the  knowledge  building 
process. 

2.  Individual  Knowledge  Building 

The  second  stage  of  the  measurement  model  focuses  on  Individual  Knowledge 
Building  (IKB).  The  cognitive  process  of  team  members  can  flow  to  IKB  or  to  team 
knowledge  building  (TKB).  IKB  actions  are  actions  such  as  reading  or  asking  questions 
that  can  be  taken  by  an  individual  to  increase  his  or  her  knowledge  (Fiore  et  ah,  in  press). 
An  individual  or  the  team  as  a  whole  asking  for  more  information  is  a  sign  that  an 
individual  or  team  members  are  trying  to  improve  individual  situational  awareness.  To 
achieve  situational  awareness,  members  must  take  immediate  action  to  correct 
information  deficiency  and  gain  knowledge.  To  increase  their  knowledge,  team  members 
can  either  ask  other  team  members  for  help  or  obtain  outside  specific  job  related 
knowledge  building  from  schools  or  training.  As  the  measurement  model  shows,  the 
feedback  loop  allows  members  to  seek  outside  knowledge  building. 

3.  Team  Knowledge  Building 

The  third  stage.  Team  Knowledge  Building  (TKB),  is  a  highly  dynamic  and 
iterative  process.  TKB  facilitates  information  exchange  among  teammates  with  the  intent 
to  generate  a  plan  or  coordinate  some  type  of  action  (Fiore  et  ah,  in  press).  Actionable 
information  will  be  processed  and  disseminated  as  a  solution  to  team  related  problems 
and  non-actionable  information  will  remain  in  the  minds  of  team  members  as  internalized 
knowledge.  A  similar  process  takes  place  within  an  AOC  targeting  cell.  Information 
received  on  a  high-value  target  may  not  be  used  to  perform  targeting  mission.  Some 
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information  will  be  incomplete  and  require  validation  through  collection  from  overhead 
or  ground  assets.  Until  further  information  is  collected,  unused  data  will  be  stored  in 
databases  until  more  information  becomes  available. 

4,  Externalized  Team  Knowledge 

The  fourth  stage,  Externalized  Team  Knowledge  (ETK)  is  defined  as  knowledge 
agreed  upon  by  team  members  to  be  factual  (Eiore  et  al,  in  press).  Knowledge  held  by 
team  members  is  different  from  information,  because  unlike  information,  agreed  upon 
knowledge  is  put  through  processes  to  ensure  accuracy  and  completeness  (Eiore  et  ah,  in 
press).  Such  processes  could  range  from  verifying  sources  to  comparing  output  from 
multiple  intelligence,  surveillance  and  reconnaissance  assets.  The  intelligence 
community  uses  a  similar  process  to  ensure  information  received  from  various 
intelligence  sources  is  accurate.  Data  collected  goes  through  a  validation  process  and  is 
analyzed  before  the  information  can  be  considered  as  intelligence. 

5.  Team  Problem  Solving  Outcomes 

During  the  Team  Problem  Solving  Outcomes,  stage  potential  solutions  are 
assessed  to  determine  if  they  meet  certain  criteria  of  effectiveness  (Eiore  et  ah,  in  press). 
In  a  naturalistic  environment,  AOC  target  cell  members  conduct  battle  damage 
assessment  to  measure  the  effectiveness  of  weapons  used.  According  to  the 
commander’s  handbook  for  Joint  Battle  Damage  Assessment,  battle  damage  assessment 
is  the  “timely  and  accurate  estimate  of  damage  resulting  from  the  application  of  military 
force,  either  lethal  or  nonlethal,  against  a  predetermined  objective.”  By  using  various 
combinations  of  intelligence,  surveillance  and  reconnaissance  assets,  commanders  can 
measure  the  performance  of  their  forces  and  weapons  effectiveness  against  different 
targets. 
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V.  METHOD 


A.  EXERCISE  DATA  SELECTION 

The  October  2008  Air  Force  Research  Event  scenarios  were  conducted  in  six  two- 
hour  sessions.  On  day  one  and  day  two,  two  sessions  were  executed  on  each  day  (one 
AM  and  one  PM  session).  On  day  three  and  day  four,  only  one  session  was  conducted, 
and  on  day  four,  exercise  personnel  were  debriefed. 

Both  authors,  in  conjunction  with  the  thesis  advisor  determined  that  the  best  data 
for  conducting  analysis  and  coding  to  empirically  evaluate  the  model  of  team 
collaboration  would  be  the  data  generated  during  the  middle  four  sessions.  Our  reasoning 
was  based  on  our  expectation  that  there  would  be  a  learning  curve  on  day  one  as 
personnel  were  becoming  familiar  with  the  exercise  operating  environment,  procedures, 
and  other  exercise  personnel  operators. 

B,  DATA  FORMATTING 

The  Air  Force  Research  Event  chat  logs  were  presented  to  us  spread  out  in  several 
tables  located  in  one  access  database  file  that  was  generated  from  the  CAOC 
Performance  Assessment  System  (CPAS),  which  was  developed  by  Johns  Hopkins 
University  Applied  Physics  Eaboratory  to  mine  data  from  AOC  systems.  Exercise  chat 
logs  included  15  different  internal  chat  rooms.  Both  authors  extracted  the  Mardem-Bey 
Internet  Relay  Chat  log  data  from  a  CPAS  database  access  file.  Data  was  pulled  from  the 
fifteen  different  chat  logs  from  (4)  two-hour  exercise  sessions  selected  for  coding.  The 
data  was  then  imported  into  a  Microsoft  excel  document  with  four  tabs,  each  labeled  and 
containing  the  data  from  the  four  sessions  selected  for  analysis.  Each  data  set  was  then 
organized  by  specific  chat  room  and  then  ordered  by  time  of  transmission  (earliest  to 
latest).  Each  chat  log  entry  also  contains  originator  and  destination  chat  room 
information. 
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C.  PRACTICE  CODING 

Practice  coding  was  completed  by  both  coders  utilizing  the  revised  set  of 
maeroeognitive  process  included  in  the  model  of  team  collaboration  on  1000  lines  of 
maritime  interdiction  operation  exercise  data.  Both  coders  began  by  coding  100  lines  of 
chat  log  data  separately  and  then  reviewed  the  eoding  with  the  thesis  advisor.  This 
initial  practiee  coding  and  review  of  additional  practice  coding  on  a  separate  transeript 
was  considered  suffieient  training  to  calibrate  the  coders  on  the  coding  process.  As  a 
team,  we  reviewed  additional  lines  of  eoding  along  with  the  definitions  and  focused  the 
discussion  on  where  coders  disagreed  on  their  coding  of  individual  chat  log  entries.  After 
this  review  of  the  initial  coded  data,  the  rest  of  the  maritime  interdietion  operation  data 
was  coded  separately  by  the  two  authors  and  then  reviewed  to  help  ensure  the  two  eoders 
were  well  ealibrated. 

D,  FINAL  CODING  OF  TRANSCRIPTS 

All  four  selected  Researeh  Event  sessions  were  coded  separately  by  the  coders. 
The  first  100  lines  from  the  Day  One  AM  data  set  were  coded  first.  Onee  eompleted,  the 
coding  was  discussed  and  reviewed  with  the  thesis  advisor  to  ensure  maeroeognitive 
process  definitions  were  being  applied  in  a  eonsistent  manner.  Following  this  final 
review  and  calibration,  both  coders  coded  the  rest  of  the  Day  One  AM  session  and 
reviewed  all  coded  data  to  ensure  a  eonsistent  interpretation  of  the  maeroeognitive 
proeess  definitions.  During  the  review,  new  codes  and  definitions  that  were  diffieult  to 
interpret/apply  were  diseussed  and  justified  to  reeonfirm  the  eoding  proeess.  The 
subsequent  remaining  three  exereise  sessions  were  coded  separately  and  then  reviewed  at 
the  eompletion  of  eaeh  session  prior  to  moving  onto  a  new  session.  Again,  this  was  done 
to  discuss  any  new  codes  assigned  and  to  discuss  the  more  difficult  speech  turns.  This 
rigorous  process  was  employed  to  ensure  the  Research  Event  team  communieations  were 
interpreted  eorrectly  and  to  help  ensure  a  eonsistent  applieation  of  the  maeroeognitive 
proeess  definitions. 


46 


E,  MEASURE  OF  INTER-RATER  RELIABILITY 

Both  authors  coded  2,493  lines  of  ehat  log  entries  contained  within  the  four 
seleeted  exereise  sessions.  Once  the  coding  was  completed,  both  coders  reviewed  the 
data  again  to  try  to  resolve  differences  in  some  codes  assigned  to  data.  After  thorough 
and  diligent  deliberation,  there  were  some  originally  assigned  codes  that  were  ehanged  to 
match  the  other  coder’s  code  but  not  all  codes  match  as  there  still  remained  some 
differences  of  opinion  between  code  definition  interpretations.  To  determine  the  overall 
percentage  of  agreement  between  the  two  coders,  the  qualitative  categorical  statistic 
Kappa  Cohen  was  used  to  caleulate  the  pereentage  of  agreement  between  the  two  coders 
using  the  eode  values  assigned  by  eaeh  eoder.  An  additional  eode  was  ereated  called  the 
Extra  Code  Filler  (ECF).  The  ECF  code  was  necessary  to  ensure  that  eaeh  eoder 
assigned  the  exact  same  number  of  total  codes  in  order  to  accurately  ealculate  inter-rater 
reliability  for  the  two  eoders,  using  Kappa  Cohen.  Kappa  Cohen  is  the  preferred  statistic 
over  the  Chi-square  statistie  as  Kappa  Cohen  tests  for  agreement  where  as  Chi-square 
tests  for  assoeiation  (Thomas,  &  Hersen,  2003).  Kappa  Cohen  is  a  better  statistic  for 
measuring  categorical  items  as  it  aceounts  for  and  factors  into  the  caleulation  that  eaeh 
eoder  may  also  agree  by  chanee  and  not  strictly  because  they  ehose  the  same  seleetion 
option  or  code.  Figure  17  shows  the  Kappa  Cohen  statistie.  In  the  Kappa  Cohen 
equation,  Pr(a)  value  is  the  observed  agreement  among  coders  and  Pr(e)  is  the 
hypothetical  probability  of  chance  agreement. 

Pr(a)-Pr(e) 

1-Pr(e) 

Figure  17.  Kappa  Cohen  Statistical  Equation  (From  Wikipedia,  2009). 

Kappa  Cohen  has  a  range  of  0  to  1 .  The  larger  the  value  calculated  indicates  a 
greater  agreement  between  the  two  coders.  A  Kappa  Cohen  value  of  .0  to  .20  indieates  a 
slight  agreement,  .21  to  .40  indicates  a  fair  agreement,  .41  to  .60  indicates  a  moderate 
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agreement,  .61  to  .80  a  substantial  agreement,  and  .81  to  1  indieates  an  almost  perfect 
agreement.  A  Kappa  Cohen  value  of  0  indicates  no  agreement  between  coders 
(Wikipedia,  2009). 

To  calculate  Kappa  Cohen,  both  coders  organized  the  exercise  data  so  there  was 
only  one  code  assigned  to  each  line  of  chat  log  text.  If  one  coder  did  not  code  a  specific 
line  of  data,  that  is,  if  there  was  a  disagreement  between  the  coders  on  whether  a  code 
should  have  been  assigned,  the  ECF  code  was  used  to  ensure  each  coder  assigned  the 
same  total  number  of  codes.  The  macrocognitive  process  definitions  were  assigned  a 
numerical  value  for  each  code  (i.e..  Administrative  =  1,  Miscellaneous  =  2,  Team 
Information  Exchange  =  3,  etc.)  and  then  a  15x15  contingency  table  was  filled  out  using 
the  coder  assigned  codes.  A  15x15  table  was  needed  and  used  during  our  coding  process 
as  we  assigned  15  different  codes  to  our  data.  Figure  18  shows  a  sample  3x3  contingency 
table  to  illustrate  the  calculation  process: 


Rater  #1 


Rater  #2 

"y" 

ft  ^  I? 

"c" 

"y" 

9 

3 

1 

II  It 

4 

8 

2 

"c" 

2 

1 

6 

Figure  18.  Sample  3x3  Contingency  Table  (From  University  Nebraska  2009). 

In  Figure  18,  the  diagonal  cells  of  the  matrix  indicate  agreement  between  the 
coders  whereas  the  other  cells  and  associated  values  indicate  the  difference  between  what 
each  of  the  coders  chose.  In  the  matrix,  both  raters  agreed  9  times  on  “y”,  8  times  on 
“r”,  and  6  times  on  “c”.  By  totaling  column  1  and  row  1,  you  can  deduce  that  rater  1 
selected  “y”  15  times  and  rater  2  chose  “y”  13  times.  By  totaling  all  the  column  values 
or  rows  you  can  find  the  total  number  of  codes  assigned  per  rater.  In  the  matrix  above, 
36  codes  were  assigned. 
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To  determine  Pr(a)  in  the  Kappa  Cohen  equation  for  the  above  matrix,  one  takes 
the  diagonal  values  9,  8,  and  6  (sum  23)  and  divides  by  36,  whieh  is  the  total  number  of 
eodes  assigned,  where  the  result  is  Pr(a)  =  .6389.  To  find  Pr(e),  one  must  determine  the 
pereentage  of  times  rater  1  and  rater  2  ehose  “y”,  “r”  and  “c”  individually  and  multiply 
the  “y”,  “r”,  and  “e”  pereentages  against  one  another  and  sum  the  results.  For  the  above 
matrix,  rater  1  ehose  “y”  15  times  out  of  36,  resulting  in  .4167  pereent,  “r”  12  times  out 
36,  resulting  in  .3333  pereent,  and  “e”  9  times  out  of  36,  resulting  in  .25  pereent.  Rater  2 
ehose  “y”  13  times  out  36,  resulting  in  .3611  pereent,  “r”  14  times  out  36,  resulting  in 
.3889  percent,  and  “c”  9  times  out  of  36,  resulting  in  .25  percent.  Rater  1  “y”  times  rater 
2  “y”  =  .1505.  Rater  1  “r”  times  rater  2  “r”  =  .1296.  Rater  1  “c”  times  rater  2  “c”  = 
.0625.  By  adding  up  the  percentages  .1505,  .1296,  and  .0626,  Pr(e)  equal  .3426. 
Therefore,  k  is  calculated  as  .4507,  which  indicates  rater  1  and  rater  2  have  a  moderate 
inter-rater  reliability  of  agreement. 
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VI.  RESULTS 


A,  CODING  RESULTS 

1,  Code  Definitions  and  Interpolation 

The  Research  Event  was  analyzed  and  coded  as  four  individual  event  sessions. 
The  analyzed  classified  Appendix  1  data  is  retained  in  the  classified  section  of  the  Naval 
Postgraduate  School  Dudley  Knox  Library.  The  first  session’s  data  contained  528  entries 
of  text,  session  two  through  contained  625,  668,  and  672  entries  of  text,  respectively. 
Total  entries  of  text  analyzed  and  assigned  codes  were  2,493.  Session  text  consisted  of 
anywhere  from  a  1  word  statement  to  a  40  word  paragraph.  Most  entries,  based  on 
macrocognitive  complexity  and  number  of  utterances,  were  assigned  multiple  codes.  For 
the  2,493  lines  of  text  analyzed,  3,158  codes  were  assigned  by  each  coder.  These  codes 
represent  the  cognitive  processes  used  during  the  Research  Event.  Each  coder  analyzed 
the  text  one  line  at  a  time  and  assigned  a  code  for  each  line  of  text.  This  process  was 
necessary,  along  with  the  use  of  the  Extra  Code  Filler  codes  to  ensure  equal  distribution 
of  codes  between  coders  when  there  was  a  disagreement  on  whether  a  code  should  or 
should  not  be  assigned,  to  facilitate  the  calculation  of  inter-rater  reliability.  Research 
Event  coded  speech  turn  examples  are  listed  in  Table  4  along  with  their  associated 
measurement  model  code  definition. 

Table  4.  Measurement  Model  Macrocognitive  Process  Code  Definitions  and 

Research  Event  Coded  Examples. 


1 

stage  I:  Internalized  Team  Knowledge  Process:  Refers  to  the  collective  knowledge  held 

in  the  individual  minds  of  team  members.  Internalized  Team  Knowledge  is  measured  by  eliciting  it 

from  individual  team  members  using  methods  such  as  card  sorting,  concept  mapping,  paired 

comparison  ratings,  scenario  probes. 

Team  Knowledge  Similarity:  Team  knowledge  similarity  can  involve  the  degree  to  which 

differing  roles  understand  one  another  (e.g.,  how  well  a  land/sea  vehicle  specialist  understands  a 

humanitarian  specialist),  or  how  well  the  team  members’  understand  the  critical  goals  and  locations 

of  important  resources  (shared  situation  awareness). 

-  No  Coded  examples  for  AOC  data 
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Team  Knowledge  Resources:  Team  members’  collective  understanding  of 

resources/responsibilities  associated  with  the  task. 

- 1  remember  sketchy  authentication  codes 

-  Fighter  aircraft  #2  is  out  of  position,  looks  like  other  strike  assets  are  quicker 

-  Fie  wouldn ’t  request  to  return  to  base  (RTB),  he  tells  you  he  is  RTB 

2 

Staae  II:  Individual  Knowledge  Building  Process:  is  a  process  which  includes  actions 

taken  by  individuals  in  order  to  build  their  own  knowledge.  These  processes  can  take  place  inside 

the  head  (e.g.,  reading,  mentally  rotating  objects)  or  may  involve  overt  actions  (e.g.,  accessing  a 

screenshot). 

Individual  Information  Gathering:  Individual  information  gathering  involves  actions 

individuals  engage  in  to  add  to  their  existing  knowledge  such  as  reading,  asking  questions,  accessing 

displays,  etc. 

-  What  is  the  correct  way  to  pass  tasking  to  a  predator  to  attack? 

-  Joint  coordinating  elements  do  you  know  the  local  threat/  risk  in  the  area 

and  do  you  have  imagery  of  the  locations? 

-  Any  battle  damage  assessment  reports/imagery  post-strike  for  aircraft? 

Individual  Information  Synthesis:  Individual  information  synthesis  involves  comparing 

relationships  among  information,  context,  and  artifacts  to  develop  actionable  knowledge. 

-  Reliable  sources  report  a  known  country  bomb  component  supplier  is 

awaiting  a  large  shipment  of  explosives 

-  It  is  suspected  that  a  certain  country  uses  this  location  as  a  storage  facility 

for  spent  fuel. 

Knowledge  Object  Development:  Knowledge  object  development  involves  creation  of 

cognitive  artifacts  that  represent  actionable  knowledge  for  the  task. 

-  No  Coded  examples  for  AOC  data 

3 

Stage  III:  Team  Knowledge  Building  Process:  is  a  process  which  includes  actions  taken 

by  teammates  to  disseminate  information  and  to  transform  that  information  into  actionable 

knowledge  for  team  members. 

Team  Information  Exchange:  Team  information  exchange  involves  passing  relevant 

information  to  the  appropriate  teammates  at  the  appropriate  times. 

-  Target  priority  coordinated,  entered  and  pushed  to  joint  time  sensitive 

targeting  manager 

-  The  actual  snatch  and  grab  would  be  possibility  for  special  operation 

force  (SOF)  but  we  would  need  intelligence  assistance 

For  your  information,  this  area  is  now  under  SOF  control.  Reconnaissance 

aircraft  to  provide  over  watch,  SOF  is  now  in  contact  with  aircraft 
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Team  Knowledge  Sharing:  Team  knowledge  sharing  involves  explanations  and 

interpretations  shared  between  team  members  or  with  the  team  as  a  whole. 

-  Self  defense  applies  for  hostile  acts  from  one  country  airspace  to  another 

-  Enemy  forces  that  employ  ordnance,  electronic  attack  or  achieve  a  radar 

lock  against  friendly  forces  have  committed  a  hostile  act. 

Team  Solntion  Option  Generation:  Team  solution  option  generation  deseribes  offering 

potential  solutions  to  a  problem. 

-  Awaiting  radiological  impact  assessment  on  watershed  if  strike  building. 

Second  option  in  work  is  destroy  local  roads  to  prevent  access  in/out. 

-  If  we  crater  the  runway  and  taxiways,  we  may  be  able  to  effectively  stop  the  target. 

-  To  shorten  timeline  for  tactical  tomahawk  we  can  launch  to  loiter. 

will  attempt  to  mitigate  with  weaponeering 

Team  Evaluation  and  Negotiation  of  Alternatives:  Team  evaluation  and  negotiation  of 

alternatives  deseribes  elarifying  and  diseussing  the  pros  and  eons  of  potential  solution  options. 

-  Just  throwing  this  out  there,  but  if  you  target  the  roadways,  is  there  a 

chance  you  could  spook  them  and  they  might  fire  off  their  missiles  and  run? 

4 

Stage  IV:  Externalized  Team  Knowledge  Process:  Refers  to  faets,  relationships,  and 

eoneepts  that  have  been  explieitly  agreed  upon,  or  not  openly  ehallenged  or  disagreed  upon,  by 

faetions  of  the  team. 

Externalized  Cue-strategy  Associations:  Externalized  eue-strategy  assoeiations  deseribe 

the  team’s  eolleetive  agreement  as  to  their  task  strategies  and  the  situational  eues  that  modify  those 

strategies  (and  how). 

-  The  dynamic  effect  cell  chief  stated  that  if  there  is  an  erect  launcher  in  a 

joint  special  operations  area  the  "rule  of  engagement  ”  is  to  kill  it  as  soon 

as  possible  and  if  there  is  time  to  de-conflict  with  the  teams 

He  mentioned  tomahawk  land  attack  missile  (TEAM)  wouldn't  be  de-confiicted 

either,  but  I  dispute  that  logic.  First,  we  wouldn 't  use  a  TEAM  shot  to  kill  a  launcher 

I  don 't  think.  Unless  it  was  a  last  resort. 

-  Can  get  special  operation  force  Team  to  location  as  additional  resource  if  we  elect 

to  monitor  the  site  for  any  potential  leadership  meetings  that  may  occur  later 

Pattern  Recognition  and  Trend  Analysis:  Pattern  reeognition  and  trend  analysis  is  the 

aeeuraey  of  the  patterns  or  trends  explieitly  noted  by  members  of  a  team  that  is  either  agreed  upon  or 

unehallenged  by  other  team  members. 

-  Eooks  like  this  target  may  be  similar  to  our  first  target  with  regards  to  unknown 

presence  of  Radiological  containers  in  facility.  We  would  look  at  interdiction  for 

containment  to  prevent  travel  to/fin  that  site,  your  thoughts  on  best  plan/option 
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Uncertainty  Resolntion:  Uncertainty  resolution  is  the  degree  to  which  a  team  has 

collectively  agreed  upon  the  status  of  problem  variables  (e.g.,  hostile/friendly). 

-  Tomahawk  land  attack  missile  most  definitely  have  to  be  de-confiicted  even  for  over 

flight  of  the  joint  special  operations  area  unless  direct  otherwise  by  the  Joint  Force 

Commander 

5 

Staae  V:  Team  Problem  Solvina  Outcomes:  Are  assessments  of  aualitv  relating  to  a 

team’s  problem  solutions  or  plan. 

-  No  Coded  examples  for  AOC  data 

Quality  of  Plan:  Quality  of  plan  (problem  solving  solution)  involves  the  degree  to  which 

the  solution  adopted  by  a  problem  solving  team  achieves  a  resolution  to  the  problem  (e.g.,  limit 

fatalities,  limit  destruction). 

-  No  Coded  examples  for  AOC  data 

Efficiency  of  Planning  Process:  Efficiency  of  planning  process  describes  the  amount  of 

time  it  takes  a  problem  solving  team  to  arrive  at  a  successful  resolution  to  a  problem. 

-  No  Coded  examples  for  AOC  data 

Efficiency  of  Plan  Execution:  Efficiency  of  plan  execution  describes  the  quality  of  the 

plan  (e.g.,  number  of  lives  saved)  divided  by  the  amount  of  resources  used  to  accomplish  this  and  the 

amount  of  time  the  plan  takes  to  unfold. 

-  No  Coded  examples  for  AOC  data 

During  our  coding  of  the  data,  both  coders  came  across  certain  speech  turns  that 
did  not  adequately  represent  one  of  the  definitions  in  the  measurement  model.  To 
properly  represent  the  data  via  assigning  codes,  it  was  necessary  for  us  to  use  non¬ 
measurement  model  codes  to  represent  certain  speech  turns.  Table  5  lists  the  non¬ 
measurement  model  codes  and  associated  definitions  that  were  necessary  to  cover  gaps 
between  the  measurement  model  codes  and  the  session  data. 
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Table  5.  Non-Measurement  Model  Codes/Definitions  and  Air  Foree  Researeh 

Event  Coded  Examples. 


Administration:  Codes  necessary  for  exercise  support  but  not  relevant  or  pertinent  to  the  exercise 
scenarios. 

-  Test 

-  In  the  future,  please  post  reports  in  “InteTReport  ” 

-  Cancel  that 

-  Command  and  control  information  posted  to  folder 

-  Chat  check 

Miscellaneous:  Codes  that  did  not  include  a  maeroeognitive  process  but  were  part  of  normal 
closed-loop  communications. 

-  Copy,  please  note  request  for  information 

-  Roger  on  target  location 
-Roger,  thank  you 

-  SIDO,  standby,  checking 

-  Copy  and  standby  far  additional  informational 

Course  of  Action:  Action  given  that  when  implemented  will  significantly  affect  the  scenario 
outcome. 

-  Contact  fighter  aircraft  #12  on  circuit  #2  for  clearance  to  drop  weapons. 

-  Move  aircraft  to  provide  over  watch  for  Special  Operation  Force  teams 

-  You  can  move  fighter  aircraft  #2  and  #10  to  training  camp  located  in  the  vicinity  of 
of  city  #3  and  city  #5.  Upon  completion  of  mission,  return  to  current  location 

-  Plan  is  to  strike  unless  directed  otherwise 

-  Move  aircraft  to  investigate  lED  implantation  report 

4 

Reauest  to  Take  Action:  Lower-level  action  request  between  peers  that  most  likelv  would  not 
affect  the  scenario  outcome. 

-  Please  instruct  aircraft  #1  to  observe  possible  SCUD  hiding  site 

-  Please  pass  report  to  all 

-  Need  you  to  check  with  air  combatant  commander  and  special  operations 
commander  for  teams  in  area 

-  Recommend  kinetic  destruction  target 

2,  Percentage  of  Codes 

The  measurement  model  for  maeroeognitive  researeh  ineludes  18  maeroeognitive 
proeess  definitions  whieh  faeilitate  the  eategorization  and  measurement  of 
maeroeognition  demonstrated  in  teams.  Analysis  of  the  Air  and  Spaee  Operations  Center 
(AOC)  dynamie  effeets  eell  (DEC)  eommunieations  revealed  that  during  the  Oetober 
Researeh  Event,  13  out  of  18  measurement  model  maeroeognitive  proeesses  were  used 
during  the  exereise.  Additional  non-measurement  eodes  sueh  as  Administrative 
(ADMIN),  Miseellaneous  (MISC),  Course  of  Aetion  (COA)  and  Request  Take  Aetion 
(RTA)  were  assigned  to  eover  gaps  in  the  measurement  model  eodes  and  our  Researeh 
Event  data.  Table  6  presents  pereentage  of  maeroeognitive  proeesses  ineluding 
administrative,  miseellaneous,  and  Extra  Code  Eiller  eodes. 
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Table  6. 


Macrocognitive  Process  Code  Percentages  including  Administrative, 
Miscellaneous  and  Extra  Code  Filler. 


Code 

Macrocognitive  Process  Categories 

Individual  Knowledge  Building 

Coder  1 # 

Coder  2# 

Coder  1  % 

Coder  2  % 

MG 

Individual  Information  Gathering 

537 

526 

17.00 

16.66 

IIS 

Individual  Information  Synthesis 

72 

33 

2.28 

1.04 

KOB 

Knowledge  Object  Development 

0 

0 

0.00 

0.00 

Team  Knowledge  Building 

TIE 

Team  Information  Exchange 

1192 

1187 

37.75 

37.59 

TKS 

Team  Knowledge  Sharing 

209 

172 

6.62 

5.45 

TSOG 

Team  Solution  Option  Generation 

19 

11 

0.60 

0.35 

TENA 

Team  Evaluation  and  Negotiation  of  Alternatives 

12 

4 

0.38 

0.13 

TPPR 

Team  Process  and  Plan  Regulation 

0 

0 

0.00 

0.00 

Internalized  Team  Knowledge 

ITK-TKS 

Team  Knowledge  Similarity 

2 

1 

0.06 

0.03 

ITK-TKR 

Team  Knowledge  Resources 

4 

2 

0.13 

0.06 

IK 

IK  -  Interpositional  Knowledge 

3 

1 

ISA 

ISA  -  Individual  Situational  Awareness 

1 

1 

Externalized  Team  Knowledge 

ECSA 

Externalized  Cue-Strategy  Association 

12 

4 

0.38 

0.13 

PRTA 

Pattern  Recognition  and  Trend  Analysis 

4 

0 

0.13 

0.00 

UR 

Uncertainty  Resolution 

1 

0 

0.03 

0.00 

Problem  Solving  Outcomes 

OOP 

Quality  of  Plan 

0 

0 

0.00 

0.00 

EPP 

Efficiency  of  Planning  Process 

0 

0 

0.00 

0.00 

EPE 

Efficiency  of  Planning  Execution 

0 

0 

0.00 

0.00 

Decision  to  Take  Action 

COA 

Course  of  Action 

154 

149 

4.88 

4.72 

RTA 

Request  Take  Action 

81 

87 

2.56 

2.75 

Administrative,  Miscellaneous,  Statistical 

MISC 

Miscellaneous  Actions/Comments 

666 

670 

21.09 

21.22 

ADMIN 

Administrative  Actions/Comments 

185 

185 

5.86 

5.86 

ECF 

Extra  Code  Filler 

8 

127 

0.25 

4.02 

100.00 

100.00 

Table  6  presents  the  recalculated  percentages  using  each  coder’s  individual  code 
assignments  and  using  Extra  Code  Filler  codes  which  ensured  both  coders  assigned  the 
exact  same  number  of  codes.  Table  7  percentages  were  calculated  using  only  the  coder’s 
individual  code  assignments.  Coder  1  assigned  2,299  codes  and  coder  2  assigned  2,176 
codes. 
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Table  7. 


Macrocognitive  Process  Code  Percentages  excluding  Administrative, 
Miscellaneous  and  Extra  Code  Filler. 


Code 

Macrocognitive  Process  Categories 

Individual  Knowledge  Building 

Coder  1 # 

Coder  2  # 

Coder  1  % 

Coder 

2% 

MG 

Individual  Information  Gathering 

537 

526 

23.36 

24.17 

IIS 

Individual  Information  Synthesis 

72 

33 

3.13 

1.52 

KOB 

Knowledge  Object  Development 

0 

0 

0.00 

0.00 

Team  Knowledge  Building 

TIE 

Team  Information  Exchange 

1192 

1187 

51.85 

54.55 

TKS 

Team  Knowledge  Sharing 

209 

172 

9.09 

7.90 

TSOG 

Team  Solution  Option  Generation 

19 

11 

0.83 

0.51 

TENA 

Team  Evaluation  and  Negotiation  of 
Alternatives 

12 

4 

0.52 

0.18 

TPPR 

Team  Process  and  Plan  Regulation 

0 

0 

0.00 

0.00 

Internalized  Team  Knowledge 

ITK- 

TKS 

Team  Knowledge  Similarity 

2 

1 

0.09 

0.05 

ITK- 

TKR 

Team  Knowledge  Resources 

4 

2 

0.17 

0.09 

IK 

IK  -  Interpositional  Knowledge 

3 

1 

ISA 

ISA  -  Individual  Situational  Awareness 

1 

1 

Externalized  Team  Knowledge 

ECSA 

Externalized  Cue-Strategy  Association 

12 

4 

0.52 

0.18 

PRTA 

Pattern  Recognition  and  Trend  Analysis 

4 

0 

0.17 

0.00 

UR 

Uncertainty  Resolution 

1 

0 

0.04 

0.00 

Problem  Solving  Outcomes 

OOP 

Quality  of  Plan 

0 

0 

0.00 

0.00 

EPP 

Efficiency  of  Planning  Process 

0 

0 

0.00 

0.00 

EPE 

Efficiency  of  Planning  Execution 

0 

0 

0.00 

0.00 

Decision  to  Take  Action 

COA 

Course  of  Action 

154 

149 

6.70 

6.85 

RTA 

Request  Take  Action 

81 

87 

3.52 

4.00 

100.00 

100.00 

Table  7  reflects  overall  code  percentages  after  removing  the  extraneous  codes 
administrative,  miscellaneous  and  extra  code  filler  codes.  Administrative  and 
miscellaneous  communications  were  prevalent  throughout  the  exercise  but  do  not 
represent  a  cognitive  process  therefore  that  type  of  communications  falls  outside  the 
scope  of  analysis  for  this  thesis.  The  most  frequently  assigned  codes,  as  shown  in  Table 
6  were  Team  Information  Exchange,  37  percent,  miscellaneous,  21  percent,  and 
Individual  Information  Gathering,  approximately  17  percent.  Table  7  shows  the 
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recalculated  percentages  of  the  macrocognitive  processes  used  when  the  administrative, 
miscellaneous,  and  ECF  codes  were  removed  from  the  calculations.  Team  Information 
Exchange,  53  percent,  and  Individual  Information  Gathering,  24  percent,  had  the  highest 
usage.  After  removing  the  extraneous  codes,  the  Decision  to  Take  Action  (combination 
of  Course  of  Action  and  Request  Take  Action  codes)  encompasses  approximately  11 
percent  of  the  communications  and  Team  Knowledge  Sharing  makes  up  about  8  percent 
of  coded  communications. 

3,  Code  Trends 

One  of  our  main  assumptions  before  analyzing  and  coding  the  Research  Event 
data  was  that  there  would  be  a  need  for  a  tremendous  amount  of  information  sharing 
amongst  internal  and  external  team  members  and  a  lot  of  information  gathering  by 
individual  team  members  required  to  appropriately  fix,  find,  track,  target,  engage  and 
asses  dynamic  targets.  After  removing  the  extraneous  codes  assigned  to  the  dynamic 
targeting  center  data,  the  percentages  (shown  in  Table  7)  indicate  that  approximately  53 
percent  of  the  data  was  coded  as  Team  Information  Exchange  and  approximately  24 
percent  was  coded  as  Individual  Information  Gathering.  Roughly  77  percent  of  the  coded 
data  is  attributed  to  information  gathering  or  information  relay  which  validated  our  main 
assumption  of  large  amounts  of  information  sharing  and  passing. 

Another  assumption  that  we  had  regarding  the  data  is  that  more  information 
gathering  would  take  place  earlier  in  the  exercise  in  order  for  individuals  to  build  an 
initial  mental  picture  and  become  familiar  with  exercise  and  associated  scenario 
parameters.  Table  8  shows  the  number  of  Individual  Information  Gathering  codes 
assigned  across  the  four  day  exercise.  Contrary  to  our  original  assumption,  exercise  day 
4  produced  the  most  Individual  Information  Gathering  codes  of  approximately  160 
compared  to  a  previous  high  of  roughly  135  codes  that  was  assigned  to  exercise  day  2 
data. 
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Table  8. 


Individual  Information  Gathering  Day  1-4  Totals. 


Individual  Information  Gathering 

Coder  1 

Coder  2 

Day  #1 

107 

103 

Day  #2 

139 

133 

Day  #3 

131 

131 

Day  #4 

160 

159 

Total 

537 

526 

After  studying  the  Researeh  Event  indoctrination  and  exercise  preparation  guide 
and  further  analyzing  the  data  on  day  4,  we  attributed  the  higher  coding  percentage  to  be 
a  result  of  multiple  dynamic  scenarios  being  run  on  day  4.  This  resulted  in  the 
participants  needing  to  clarify  and  gather  more  information  than  on  the  previous  three 
days.  Additionally,  the  Research  Event  planners  did  a  solid  job  on  preparing  the  exercise 
participants  by  conducting  an  INCHOP  brief  and  reviewing  the  Research  Event 
indoctrination  and  exercise  preparation  guide  with  all  participants  prior  to  the  start  of  the 
exercise.  This  resulted  in  less  need  for  the  participants  to  share  and  gather  initial 
information.  Table  9  shows  Team  Information  Exchange  from  day  I  compared  against 
day  4  which  also  supports  this  claim. 

Table  9.  Team  Information  Exchange  Day  I  and  Day  4  Totals. 


Team  Information  Exchange 

Coder  1 

Coder  2 

Day  #1 

225 

235 

Day  #4 

313 

317 

Another  assumption  was  that  there  would  be  more  Decisions  to  Take  Action 
codes  (combination  Course  of  Action  and  Request  Take  Action  codes)  occurring  later  in 
the  Research  Event.  This  initial  assumption  was  based  on  the  way  most  typical  exercises 
are  developed  to  build  upon  a  climatic  end  point  requiring  several  decisions  to  be  made 
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typically  near  the  end  of  the  exereise.  Table  10  lists  the  number  of  Course  of  Aetion  and 
Request  Take  Aetions  eodes  assigned  throughout  the  four  day  exereise. 


Table  10.  Course  of  Aetion  and  Request  Take  Aetion  Day  1  through  Day  4  Totals. 


Course  of  Action 

Coder  1 

Coder  2 

Day  #1 

23 

20 

Day  #2 

42 

43 

Day  #3 

42 

42 

Day  #4 

47 

44 

Total 

154 

149 

Request  Take  Action 

Coder  1 

Coder  2 

Day  #1 

15 

13 

Day  #2 

27 

34 

Day  #3 

22 

23 

Day  #4 

17 

17 

Total 

81 

87 

The  code  numbers  indicate  that  scenarios  on  day  one  and  day  four  required  the 
least  decision  making  and  that  day  2  and  day  3  consistently  required  approximately  the 
same  amount  of  Course  of  Aetion  and  Request  Take  Aetion. 

4,  New  Codes  and  Modifying  Definitions 

During  the  praetiee  coding  process,  it  was  determined  by  both  coders  that  the  new 
set  of  macrocognitive  process  definitions  ineluded  in  the  model  failed  to  appropriately 
address  and  speeifically  define  all  the  eognitive  proeesses  for  the  data  that  was  analyzed. 
Speeifieally,  there  were  no  macrocognitive  proeess  definitions  under  the  new  set  of 
definitions  that  appropriately  elassified  or  defined  exereise  personnel’s  deeision  to  take 
aetion.  Deeisions  to  Take  Aetion  were  elassified  into  two  sub-eategories,  Course  of 
Aetion  and  Request  Take  Action.  Course  of  Aetion  was  assigned  to  a  speech  turn  that 
issued  an  order  for  a  more  signifieant  aetion  that  would  be  more  likely  to  affeet  the 
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overall  scenario  outcome.  An  order  usually  was  issued  from  a  more  key  position  or 
senior  person  down  to  a  lower  level  position  or  subordinate.  Request  Take  Action  was 
assigned  to  speech  turns  that  were  lower-level  requests  between  peers  to  take  some 
action,  but  would  not  likely  affect  the  entire  outcome  of  the  scenario.  Other  codes  that 
were  assigned  but  not  part  of  the  set  of  macrocognitive  process  definitions  are 
miscellaneous  and  administrative.  Miscellaneous  codes  were  assigned  to  speech  turns 
that  did  not  include  a  macrocognitive  process  but  were  part  of  normal  closed-loop 
communications  such  as  “Roger”.  Administrative  codes  were  assigned  to  speech  turns 
that  were  necessary  for  exercise  support  but  not  relevant  or  pertinent  to  the  exercise 
scenarios  such  as  communications  checks  prior  to  start  of  the  exercise. 

B,  INTER-RATER  RELIABILITY 

Table  1 1  is  a  pivot  table  that  compares  coder  1  codes  against  coder  2  codes.  Coder 
1  codes  are  read  down  and  coder  2  codes  are  read  across.  Coder  matches  run  diagonally 
through  the  pivot  table  starting  with  code  Administrative  183,  Miscellaneous  663,  Team 
Information  Exchange  1134,  Individual  Information  Gathering  521,  Individual 
Information  Synthesis  23,  Externalized  Cue-Strategy  Association  4,  Team  Knowledge 
Sharing  136,  Course  of  Action  138,  Request  Take  Action  78,  Team  Solution  Option 
Generation  6,  Team  Evaluation  and  Negotiation  of  Alternatives  3,  Internalized  Team 
Knowledge  3,  Uncertainty  Resolution  0,  Pattern  Recognition  and  Trend  Analysis  0,  and 
Extra  Code  Eiller  0. 
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Table  11. 


Coder  Pivot  Table 


CODE 

TITLE 

ADMIN 

MISC 

TIE 

CO 

ECSA 

TKS 

COA 

< 

1 — 
a: 

TSOG 

TENA 

H 

a: 

3 

PRTA 

u. 

O 

UJ 

Total  Coder  2 

ADMIN 

183 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

1 

185 

MISC 

0 

663 

6 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

670 

TIE 

0 

1 

1134 

5 

9 

1 

24 

9 

0 

2 

0 

0 

0 

1 

1 

1187 

IIG 

2 

0 

2 

521 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

526 

IIS 

0 

0 

4 

0 

23 

0 

6 

0 

0 

0 

0 

0 

0 

0 

0 

33 

ECSA 

0 

0 

0 

0 

0 

4 

0 

0 

0 

0 

0 

0 

0 

0 

0 

4 

TKS 

0 

0 

17 

1 

6 

1 

136 

3 

0 

3 

3 

1 

0 

1 

0 

172 

COA 

0 

0 

4 

1 

0 

0 

1 

138 

1 

1 

0 

0 

0 

0 

3 

149 

RTA 

0 

0 

4 

1 

0 

0 

1 

0 

78 

0 

0 

0 

0 

0 

3 

87 

TSOG 

0 

0 

1 

0 

1 

2 

0 

0 

0 

6 

1 

0 

0 

0 

0 

11 

TENA 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

3 

0 

0 

0 

0 

4 

ITK 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

3 

0 

0 

0 

3 

UR 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

PRTA 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

ECF 

0 

2 

20 

8 

33 

4 

39 

3 

2 

6 

5 

2 

1 

2 

0 

127 

Total 
Coder  1 

185 

666 

1192 

537 

72 

12 

209 

154 

81 

19 

12 

6 

1 

4 

8 

3158 

The  pivot  table  also  illustrates  what  codes  the  coders  disagreed  upon.  Under  the 
Administrative  (ADMIN)  category,  coder  1  and  coder  2  both  had  assigned  a  total  of  185 
administrative  codes  to  the  data.  However,  both  coders  only  matched  selections  for  183 
out  185  administrative  codes.  There  were  two  disagreements  per  each  coder.  Reading 
down  the  column  under  ADMIN  you  can  see  that  coder  2  had  selected  two  Individual 
Information  Gathering  (IIG)  codes  when  coder  1  had  assigned  administrative  codes. 
Additionally,  reading  across  the  row  for  category  ADMIN,  you  can  see  that  coder  I 
selected  1  Team  Knowledge  Similarity  (TKS)  and  1  Extra  Code  Filler  (ECF)  code  when 
coder  2  had  assigned  an  ADMIN  code.  Reasons  for  disagreement  between  coders  can  be 
attributed  to  a  particular  line  of  text  not  being  interpreted  the  same  way 
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between  the  eoders  (i.e.,  eoder  1  thought  the  line  of  text  related  to  the  aetual  exereise 
whereas  eoder  2  elassified  it  as  non-essential  eommunieations  assigning  an  administrative 
eode). 

One  thing  that  stands  out  after  going  through  eaeh  eode  eategory  above  and 
eomparing  the  eommon  differenees  between  eoders  (i.e.,  when  one  eoder  ehose  X  the 
other  eoder  ehose  Y)  is  that  when  there  was  a  disagreement  on  Team  Information 
Exehange  (TIE)  or  Team  Knowledge  Sharing  (TKS)  eodes,  the  other  eode  typieally 
seleeted  eode  was  Team  Knowledge  Sharing  and  Team  Information  Exehange 
(respeetively).  Reading  down  the  pivot  table  for  eategory  TIE,  you  ean  see  that  the 
number  of  agreed  Team  Information  Exehange  eodes  is  1,134  and  that  eoder  2  disagreed 
with  eoder  1  and  seleeted  Team  Knowledge  Similarity  17  times.  Additionally,  reading 
down  Team  Knowledge  Sharing  (TKS)  eategory,  both  eoders  agreed  136  times  but  eoder 
2  disagreed  and  seleeted  Team  Information  Exehange  24  times.  The  disagreement 
between  TKS  and  TIE  and  the  patterned  alternative  response  of  the  other  eoder  eode 
indieates  that  there  is  some  ambiguity  in  the  measurement  model  definition  for  both 
eodes.  Eurthermore,  when  eoder  1  seleeted  Team  Information  Exehanges  and  Team 
Knowledge  Sharing,  eoder  2  disagreed  and  seleeted  the  Extra  Code  Eiller  (ECE)  eode  20 
and  39  times  (respeetively).  This  disagreement  between  eoders  and  non  seleetion  of  the 
measurement  model  eode  indieates  that  the  definitions  for  Team  Knowledge  Sharing  and 
Team  Information  Exehange  need  to  be  modified  to  remove  ambiguity  and  vagueness. 

1,  Kappa  Cohen  Statistic  Analysis 

Kappa  was  ealeulated  using  the  equation  listed  in  Eigure  19.  Probability  of 
Agreement,  Pr(a),  is  ealeulated  by  adding  up  all  the  agreements  of  the  fifteen  eodes  used 
and  dividing  that  number  by  the  total  eodes  assigned.  Pr(a)  =  (183  +  663  +  1134  +  521  + 
23  +  4  +  136  +  138  +  78  +  6  +  3  +  3  +  0  +  0  +  0)  /  3158  =  .915769474.  Probability  of 
Agreement  Due  to  Randomness,  Pr(e),  is  ealeulated  by  multiplying  eoder  1  eategorieal 
eodes  against  eoder  2  eategorieal  eodes  (i.e.,  ADMIN  *  ADMIN,  MISC  *  MISC,  ete..), 
summing  the  total  and  dividing  that  number  by  the  total  eodes  times  the  total  eodes. 
Pr(e)  =  (185  *  185  +  666  *  670  +  1192  *  1187  +  537  *  526  +  72  *  33  +  12  *  4  +  209  * 
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172+  154  *  149  +  81  *  87+  19  *  11  +  12  *  4  +  6  *  3  +  1  *  0  +  4  *  0  +  8  *  127)/  (3158 
*  3158)  =  .225355972.  Figure  19  shows  the  calculation  of  the  Kappa  Cohen  equation 
using  our  statistic  numbers. 


Pr(a)-Pr(c)  .915769474-.225355972 


1-Pr(c) 


1-.225355972 


=.891265507^.89 


Figure  19.  Kappa  Cohen  Statistical  Equation  Result. 


Our  .89  returned  Kappa  value  indicates  that  coder  1  and  coder  2  both  interpreted 
the  measurement  model  definitions  and  Research  Event  data  the  same  way.  A  returned 
kappa  (k)  value  ranging  between  .81  -  1  is  considered  an  almost  perfect  agreement 
between  coders  (Eandis  &  Koch,  1977). 


C.  COGNITIVE  PHASES 

1.  First  Stage  Macrocognitive  Phase 

Prior  to  being  assigned  to  an  AOC,  service  members  are  often  interviewed  and 
have  their  professional  records  screened  to  ensure  the  member  has  the  adequate  formal 
schooling,  training  and  experience  level  to  successfully  perform  AOC  tasks.  Based  on 
the  type  of  training  and  level  of  experience,  team  members  assigned  to  an  actual  AOC  or 
just  participating  in  a  coordinated  exercise,  bring  different  levels  of  knowledge  that  can 
be  applied  to  various  dynamic  situations.  In  the  first  stage  of  the  measurement  model, 
this  existing  knowledge  is  referred  to  as  Internalized  Team  Knowledge.  According  to  the 
measurement  model.  Internalized  Team  Knowledge  is  the  knowledge  held  in  the  mind  of 
an  individual  team  member.  Our  coding  of  the  Research  Event  communications  data 
revealed  that  only  .15  percent  of  the  total  data  coded  was  Internalized  Team  Knowledge. 
Internalized  Team  Knowledge  consists  of  two  cognitive  process  subcategories  called 
Team  Knowledge  Similarity  and  Team  Knowledge  Resources. 

Team  Knowledge  Similarity  looks  at  how  well  team  members  in  different  jobs 
within  the  AOC  understand  each  other’s  roles  and  responsibilities.  A  generic  example  of 
Team  Knowledge  Similarity  would  be  how  well  does  the  intelligence  officer  understand 
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the  role  of  the  JAG  offieer.  Team  Knowledge  Resourees  foeuses  on  the  team’s  overall 
eollective  understanding  of  the  task  (SUMMIT  Measurement  Model  Doeument).  Coded 
Researeh  Event  eommunication  data  did  not  provide  any  definitive  reasons  that  explained 
the  team  member’s  low  usage  of  the  Internalized  Team  Knowledge  eognitive  proeess. 
The  speeifie  reason  for  the  low  usage  may  lie  in  further  explanation  of  the  Internalized 
Team  Knowledge  definition.  Aeeording  to  the  Salas  et  al,  raw  data  assoeiated  with 
internalized  knowledge  should  be  eolleeted  prior  to  the  start  of  the  exereise  on  all  team 
members  and  then  an  after  exercise  assessment  survey  should  be  given  to  see  how  well 
they  understood  their  responsibility  and  the  overall  task. 

It  is  not  clear  whether  the  above  listed  pre-  and  post-assessments  were  conducted, 
and  no  such  data  has  been  collected  or  sent  to  the  coders.  Without  the  pre-  and  post¬ 
assessment  data,  it  is  not  possible  for  us  to  ascertain  team  members  true  Internalized 
Team  Knowledge  and  understanding  of  roles  and  tasks  by  solely  looking  at  the  Research 
Event  coded  data.  It  is  not  fair  to  say  that  the  low  score  of  Internalized  Team  Knowledge 
cognitive  processes  coded  by  us  is  a  true  and  accurate  representative  of  team  members 
overall  internalize  knowledge.  However,  evaluation  of  previous  iterations  of  the 
SUMMIT  model  using  Northeast  Air  Defense  Sector  and  Eederal  Aviation 
Administration  communications  data  from  September  11,  2001  by  Euis  E.  Socias  also 
reflected  similar  low  Internalized  Team  Knowledge  usage  and  coded  scores. 

2.  Second  Stage  Macrocognition  Phase 

Individual  Knowledge  Building,  the  second  cognitive  process  in  the  measurement 
model,  focuses  on  the  actions  taken  by  team  members  to  increase  their  overall  knowledge 
of  a  given  situation.  Even  with  years  of  training  and  schools,  individuals  may  find 
themselves  in  situations  that  require  other  steps  be  taken  to  build  on  their  existing 
knowledge.  According  to  the  SUMMIT  definitions,  such  steps  could  involve  but  are  not 
limited  to  reading  and  asking  questions.  Table  12  includes  excerpts  from  the  Air  Eorce 
Research  Event  showing  team  members  engaging  in  Individual  Information  Gathering 
(IIG)  and  Individual  Information  Synthesis  (IIS).  This  includes  asking  for  clarification 
on  information  previously  passed,  such  as  the  name  of  a  missing  airfield  to  requesting 
specific  data  on  a  known  bomb  maker. 
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Table  12. 


Individual  Knowledge  Building;  Individual  Information  Gathering  and 
Individual  Information  Synthesis  Examples. 


Individual  Knowledge  Building 

Originator 

Communication 

Code 

C2DO 

SIDO:  I  missed  the  name  of  the  airfield? 

IIG 

C2DO 

SIDO:  Type  of  aireraft  we  are  looking  for  and  its  latitude  and 

longitude? 

IIG 

SIDO 

Airfield  is  loeated  at  (*Removed*);  type  of  aireraft  is  STOL  cargo 

plane. 

TIE/TIE 

SODO 

Command  control  duty  officer  (C2DO),  what  is  the  capability  to  track 

STOL  Cargo  aircraft  with  (Tac  C2)? 

IIG 

SIDO 

C2DO,  STOL  cargo  aircraft  has  departed  to  the  target;  may  operate 

between  44  and  200  knots. 

IIS 

Individual  Knowledge  Building 

Originator 

Communication 

Code 

lOT 

NTI:  What  can  you  tell  me  about  bomb  supplier  #1? 

IIG 

NTI 

Information  operations  targeteer,  bomb  supplier  #1  is  a  known  materials 

supplier;  communication  on  cover  and  will  report  any  new  intelligence 

when  available. 

IIS/TIE 

The  pereentage  of  Researeh  Event  team  member  speeeh  turns  eoded  as  Individual 
Knowledge  Building  was  16.66  pereent.  Individual  Knowledge  Building  was  the  seeond 
most  used  maerocognitive  proeess.  The  most  used  eognitive  proeess  was  Team 
Knowledge  Building.  Although  there  is  no  definitive  answer,  the  high  usage  of 
Individual  Knowledge  Building  may  have  resulted  from  the  unfamiliar  dynamie  setting 
of  the  exereise.  The  Research  Event  placed  team  members  in  a  time-compressed 
situation  while  at  the  same  time  requiring  processing  of  a  large  amount  of  information 
from  various  intelligence  sensors.  To  build  on  their  existing  knowledge  and  to  maintain 
continuous  situation  awareness  in  the  dynamic  environment,  team  members  engaged  in 
building  their  individual  knowledge  by  asking  lots  of  questions.  High  Individual 
Knowledge  Building  speech  turns  were  also  recording  in  the  Eederal  Aviation 
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Administration  and  Fire  Department  of  New  York  communieation  data  during  the 
September  11,  2001  attack.  This  was  probably  attributed  to  the  unfamiliar  and  changing 
environment  factors. 

3,  Third  Stage  Macrocognition  Phase 

The  third  cognitive  phase  of  the  measurement  model  is  Team  Knowledge 
Building.  Team  Knowledge  Building  includes  actions  taken  by  team  members  to 
disseminate  and  transform  information  into  actionable  knowledge.  In  a  dynamic 
environment,  to  be  effective  against  time-sensitive  targeting,  AOC  team  members  must 
be  ready  at  all  times  to  make  tough  decisions  in  a  time  limited  environment.  Speech 
turns  during  the  Research  Event  shows  that  team  members  were  highly  engaged  in 
information  exchange  and  sharing.  In  Table  13,  team  members  discussed  the  effects  of 
radiological  fallout  from  a  possible  strike  against  a  building.  Such  collaboration  among 
different  dynamic  cells  could  ensure  higher  situation  awareness  and  facilitates  better 
informed  decision  making. 

Table  13.  Team  Knowledge  Building:  Team  Information  Exchange  Example. 


Team  Knowledge  Building  Process 

Originator 

Communication 

Code 

DEC 

Awaiting  radiological  impact  assessment  on  watershed  if  the  building  is  to  be 

strike.  Second,  option  in  work  is  to  destroy  local  roads  to  prevent  access  in/out. 

TIE/TSOG 

DECD 

Aircraft  returns  watershed  non-issue 

TIE 

JOCJCE 

Dynamic  effect  cell,  you  have  high-value  target  on  your  dynamic  target  list. 

What  is  the  air  combat  commander  game  plan?  If  you  have  a  good  one,  I  will 

appoint  you  the  lead  but  I  think  SOF  needs  to  be  considered. 

TIE/IIG 

Eorty-three  percent  of  the  Research  Event  team  communications  were  coded  as 
one  of  the  macrocognitive  processes  that  occur  during  the  Team  Knowledge  Building 
cognitive  processing  phase.  Macrocognitive  processes  that  were  used  among  the  team 
include  Team  Information  Exchange,  Team  Knowledge  Sharing,  Team  Solution  Option 
Generation,  and  Team  Evaluation  and  Negotiation  of  Alternatives.  Most  frequently  used 
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macrocognitive  processes  used  were  Team  Information  Exehange  (37.57%)  and  Team 
Knowledge  Sharing  (5.45%).  Other  Team  Knowledge  Building  macrocognitive 
processes  used  during  the  exereise  fell  below  1  pereent.  The  high  usage  of  Team 
Information  Exchange  was  probably  due  to  the  dynamic  nature  of  the  exercise  and  the 
complexity  involved  in  engaging  time-sensitive  targets.  Table  14  is  an  excerpt  from  the 
Research  Event  communications  data  that  shows  team  members  sharing  information  on 
rules  of  engagement  and  discussing  the  effects  of  a  strike  mission  against  an  airfield. 

Table  14.  Team  Knowledge  Sharing  Example. 


Team  Knowledge  Building  Process 

Originator 

Communication 

Code 

DEC 

Self  defense  applies  for  hostile  acts  from  Country  #3  fighters  in  Country 

#  2  or  #4  airspace 

TKS 

DEC 

Enemy  forces  that  employ  ordnance,  electronic  attack  or  achieve  a  radar 

lock  against  friendly  forces  have  committed  a  hostile  act 

TKS 

TDO 

If  we  crater  the  runway  and  taxiways,  we  may  be  able  to  effectively  stop 

the  target. 

TSOG 

lOT 

Target  Duty  Officer  (TDO):  Just  throwing  this  out  there,  but  if  you  target 

the  roadways,  is  there  a  chance  you  could  spook  them  and  they  might  fire 

off  their  missiles  and  run? 

TENA 

4,  Fourth  Stage  Macrocognition  Phase 

Externalized  Team  Knowledge  refers  to  knowledge  that  has  been  agreed  upon  by 
members  of  the  team  and  is  the  fourth  maeroeognitive  phase  in  the  measurement  model. 
Under  the  Externalized  Team  Knowledge  phase,  Researeh  Event  team  members  used  all 
three  macrocognitive  proeesses;  Externalized  Cue-Strategy  Assoeiation,  Pattern 
Reeognition  and  Trend  Analysis,  and  Uneertainty  Resolution.  The  pereentage  of  speeeh 
turns  coded  as  one  of  the  macrocognitive  processes  that  occur  in  the  Externalized  Team 
Knowledge  phase  was  only.24  percent  with  ECSA  and  PRTA  being  the  most  used  out  of 
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the  fourth  stage.  Table  15  contains  excerpts  from  the  Research  Event  showing  team 
members  coming  to  an  agreement  on  the  type  of  weapon  to  use  and  the  de-confliction 
needed. 


Table  15.  Externalized  Cue  Strategy  Association  and  Pattern  Recognition  Trend 

Analysis  Examples. 


Externalized  Team  Knowledge 

Originator 

Communication 

Code 

DECSOLE 

The  dynamic  effects  cell  chief  stated  that  if  there  is  an  erect  launcher  in  a  joint 

special  operations  area  (JSOA)  his  rules  of  engagement  (ROE)  are  to  kill  it  as 

soon  as  possible  and  if  there  is  time  to  de-conflict  with  the  teams. 

ECSA 

JSOFT 

Correct,  if  per  joint  force  commander  (JFC) 

TIE 

DECSOLE 

I  can't  remember  ROE  in  the  west  for  OIF  but  I  think  it  was  something  similar. 

TIE 

DECSOLE 

He  mentioned  tomahawk  land  attack  missiles  (TEAM)  wouldn't  be  de-conflicted 

either,  but  I  dispute  that  logic.  First,  we  wouldn't  use  a  TEAM  to  shot  a 

launcher. .  .1  don't  think.  Unless  it  was  a  last  resort. 

ECSA 

SECSOLE 

Second  the  flight  time  is  great  enough  to  pass  the  warning  and  do  the  de- 

confliction. 

TIE 

JSOFT 

TEAMS  most  definitely  have  to  be  de-conflicted  even  for  over  flight  of  the  JSOA, 

unless  directed  otherwise  by  the  JFC.  He's  not  the  JFC.  If  any  issues,  let  me 

know  and  I'll  pass  up  to  the  joint  special  operation  task  force  commander  for 

discussion  with  the  JFC. 

UR/ 

ECSA 

DEC 

looks  similar  to  our  first  target  with  regards  to  unknown  presence  of  Radiological 

containers  in  facility.  We  would  look  at  interdiction  for  containment  to  prevent 

travel  to/fm  that  site,  your  thoughts  on  best  plan/option 

PRTA 

There  is  no  definitive  answer  to  explain  why  Externalized  Team  Knowledge 
communications,  the  process  where  the  team  validates  information  for  accuracy  and 
completeness  before  taking  action,  came  in  with  such  low  percentages.  Our  attempt  to 
explain  the  low  Externalized  Team  Knowledge  percentage  is  that  most  AOC  personnel 
have  years  of  experience  and  are  trained  to  do  their  job  with  little  or  no  assistance. 
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Individual  cells  and  operators  eommunioated  via  ehat  with  other  team  members  after 
most  of  the  analyses  and  final  agreement  on  information  received  was  already  deeided 
upon  or  eompleted.  Due  to  the  layout  of  the  Researeh  Event,  exereise  personnel  located 
in  one  room  with  divider  walls  between  each  position,  it  is  highly  probable  that  team 
members  eommunieated  with  eaeh  other  via  voiee  viee  utilizing  ehat  only,  which  could 
have  resulted  in  loss  of  possible  Externalized  Team  Knowledge  speeeh  turns.  Aeeording 
to  the  measurement  model  definitions,  a  voiee  reeorder  should  be  used  to  eapture  the 
exehange  of  information  between  teammates. 

5,  Fifth  Stage  Macrocognition  Phase 

The  final  maeroeognitive  phase  in  the  measurement  model  is  Team  Problem 
Solving  Outeomes.  During  the  eoding  of  the  data,  there  was  no  Team  Problem  Solving 
Outeomes  eognitive  proeesses  found  or  eoded.  Team  Problem  Solving  Outeomes  foeuses 
on  the  quality  and  speed  by  whieh  team  members  eome  up  with  viable  solutions  to 
problems  or  develop  response  plans.  Our  possible  explanation  of  why  there  were  no 
Team  Problem  Solving  Outeome  speech  turns  is  that  the  team  members  were  dealing 
with  unique  dynamie  situations  which  made  it  diffieult  to  pull  from  past  experienees  and 
possibly  whieh  eould  have  made  exereise  partieipants  feel  uneomfortable  to  suggest 
and/or  reeommend  solutions.  Additionally,  Team  Problem  Solving  Outeomes  eould  have 
been  eommunieated  via  voiee  to  team  members  loeated  in  same  room  viee  being  sent  via 
chat. 
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VII.  CONCLUSION  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

1,  Use  of  Codes 

The  use  of  the  Researeh  Event  eommunieation  data  to  evaluate  the  measurement 
model  shows  that  throughout  the  exereise  team  members  used  13  of  the  18 
maerocognitive  proeess  codes.  Codes  not  used  fall  within  three  of  the  macrocognitive 
stages  of  the  measurement  model.  Those  not  coded  include  quality  of  plan,  efficiency  of 
planning  process,  efficiency  of  planning  execution,  knowledge  object  development,  and 
team  process  and  plan  regulation.  The  fact  that  they  were  not  coded  by  us  does  not  mean 
that  they  did  not  take  place  within  the  exercise.  What  it  means  is  that  the  manner  by 
which  theses  processes  are  captured  was  not  possible  when  analyzing  the  Research  Event 
communication  chat  room  data.  In  other  words,  these  codes  not  used  by  us  require  data 
to  be  recorded,  time  stamp,  or  be  shown  graphically  so  as  to  measure  the  outcome  and 
object  development  of  the  team. 

2,  Code  Percentage  and  Kappa  Cohen  Results 

The  code  percentage  results  and  the  Kappa  Cohen  analysis  assisted  in  empirically 
evaluating  the  SUMMIT  measurement  model  for  macrocognitive  research.  The  use  of  13 
of  the  measurement  model  codes  is  evident  throughout  the  four  analyzed  exercise 
sessions.  The  high,  almost  perfect.  Kappa  Cohen  result  further  indicates  that  both  coders 
had  a  clear  interpretation  of  the  measurement  model  code  definitions  and  assigned  the 
codes  consistently  throughout  the  coding  process. 

B,  RECOMMENDATIONS 

We  recommend  trying  to  locate  or  collect  other  DTC  data  from  future  exercises  in 
addition  to  reanalyzing  past  maritime  interdiction  operation  and  9/11  Eire  Department 
data  using  the  measurement  model  for  macrocognitive  research.  Analyzing  and  coding 
other  data  sets  using  the  same  measurement  model  for  macrocognitive  research  will  allow 
side  by  side  comparisons  of  number  of  macrocognitive  codes  used  in  team  collaboration. 
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The  comparative  analysis  can  aid  in  identifying  gaps  between  the  measurement  model 
code  definitions  and  the  macrocognitive  processes  of  individuals  and  teams  during  real- 
world  scenarios,  compared  to  laboratory  experiments.  Data  could  support  modifications 
to  the  measurement  model  code  definitions  or  lead  to  new  codes  being  added  (i.e., 
Request  Take  Action,  Course  of  Action). 

Recommend  taking  past  and  future  DTC  and  maritime  interdiction  operation  data 
and  re-calculate/calculate  inter-rater  reliability  between  coders  using  Kappa  Cohen.  The 
data  generated  from  the  Kappa  Cohen  process  (i.e.,  pivot  chart  indication  of  non 
agreement  between  coder  assigned  codes)  is  extremely  useful  in  determining  if  code 
definitions  are  too  vague.  Adjustment  of  code  definitions  and  recoding  and  calculating 
inter-rater  reliability  can  be  a  metric  for  testing  definition  ambiguity  and  if  adjustment 
aided  or  worsened  the  agreement  level. 

Recommend  that  pre-exercise  coordination  be  achieved  with  the  exercise  director 
prior  to  the  next  Air  Force  Research  Event.  Pre-coordination  could  allow  for  the  use  of 
better  macrocognition  measurement  tools  and  techniques  that  would  be  instrumental  in 
capturing  more  macrocognitive  processes  information.  Devices  such  as  voice  recorders 
could  be  used  on  the  main  exercise  floor  to  capture  exercise  personnel  speech  turns  that 
were  spoken  due  to  proximity  of  other  exercise  personnel  vice  being  sent  as  text  entries 
through  Mardem-Bey  Internet  relay  chat  communications  system.  Pre-survey  and  post¬ 
surveys  could  also  be  administered  to  exercise  personnel  that  would  help  in  determining 
if  new  knowledge  was  actually  generated  and/or  produced  during  Individual  Knowledge 
Building  and  Team  Knowledge  Building  phases.  Use  of  eye  tracking  equipment  could  be 
incorporated  at  every  exercise  participant  workstation  to  track  their  eye  movement 
leading  to  information  collection  and  possible  development  of  better  command  and 
control  graphical  user  interface  systems. 
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